2023-12-12T01:29:12.968 INFO:root:teuthology version: 0.0.1.dev216+g5f3731e 2023-12-12T01:29:12.969 DEBUG:teuthology.run:Teuthology command: teuthology --name dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi --verbose --description rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} --archive /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100 --owner scheduled_dis@teuthology -- /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/orig.config.yaml 2023-12-12T01:29:13.017 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T01:29:13.107 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100 branch: wip-dis-testing-1 description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} email: idryomov@gmail.com first_in_suite: false job_id: '7488100' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-dis-testing-1 ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dis-testing-2 sha1: e7f1b79d320b95939278401b2a5bd1b491f81fe3 owner: scheduled_dis@teuthology priority: 69 repo: https://github.com/ceph/ceph-ci roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 7033 sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef sleep_before_teardown: 0 subset: 91/99 suite: rbd suite_branch: wip-dis-testing-2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci suite_sha1: e7f1b79d320b95939278401b2a5bd1b491f81fe3 targets: smithi055.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhDAJKfrLVA+K4Ad6CRbRTxf9G11RQdKVLaVWOow2atgabyCvvM6RPcIaCoqkzXKn0+HB/ochBaexdCmpXV9EQ= smithi140.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRz2LD1q4ZzlkncmnwebNaW2K130qCITj6zPV4RfGgzvT1s4SG2i00T3UnQlVQKmsySsbKYfD5Kvf2GQXBRRzs= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73 timestamp: 2023-12-12_01:13:29 tube: smithi user: dis verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 2023-12-12T01:29:13.107 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa; will attempt to use it 2023-12-12T01:29:13.108 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks 2023-12-12T01:29:13.108 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-12T01:29:13.110 INFO:teuthology.task.internal:Checking packages... 2023-12-12T01:29:13.133 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '3d2db44fa991bc355074aa9b7647f63b77201fef' 2023-12-12T01:29:13.133 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-12T01:29:13.133 INFO:teuthology.packaging:ref: None 2023-12-12T01:29:13.133 INFO:teuthology.packaging:tag: None 2023-12-12T01:29:13.133 INFO:teuthology.packaging:branch: wip-dis-testing-1 2023-12-12T01:29:13.133 INFO:teuthology.packaging:sha1: 3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-12T01:29:13.133 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-dis-testing-1 2023-12-12T01:29:13.272 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:29:13.273 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-12T01:29:13.282 INFO:teuthology.task.internal:no buildpackages task found 2023-12-12T01:29:13.283 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-12T01:29:13.292 INFO:teuthology.task.internal:Saving configuration 2023-12-12T01:29:13.312 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-12T01:29:13.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T01:29:13.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 01:18:22.406121', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJRIWD6lPSrIVvi2nf068hGzBT/JHIpexQgS+PfgeBPh'} 2023-12-12T01:29:13.370 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 01:18:22.404126', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGHe8qxF47McpakUDsXWzsKKdBU3kj59/vkB7kzTgBC3'} 2023-12-12T01:29:13.370 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-12T01:29:13.379 INFO:teuthology.task.internal:roles: ubuntu@smithi055.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2023-12-12T01:29:13.379 INFO:teuthology.task.internal:roles: ubuntu@smithi140.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2023-12-12T01:29:13.379 INFO:teuthology.run_tasks:Running task console_log... 2023-12-12T01:29:13.457 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f61d0794dc0>, signals=[15]) 2023-12-12T01:29:13.457 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-12T01:29:13.466 INFO:teuthology.task.internal:Opening connections... 2023-12-12T01:29:13.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:29:13.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:29:13.544 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:29:13.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:29:13.620 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-12T01:29:13.629 DEBUG:teuthology.orchestra.run.smithi055:> uname -m 2023-12-12T01:29:13.635 INFO:teuthology.orchestra.run.smithi055.stdout:x86_64 2023-12-12T01:29:13.635 DEBUG:teuthology.orchestra.run.smithi055:> cat /etc/os-release 2023-12-12T01:29:13.682 INFO:teuthology.orchestra.run.smithi055.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-12T01:29:13.682 INFO:teuthology.orchestra.run.smithi055.stdout:NAME="Ubuntu" 2023-12-12T01:29:13.682 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_ID="22.04" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:VERSION_CODENAME=jammy 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:ID=ubuntu 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:ID_LIKE=debian 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-12T01:29:13.683 INFO:teuthology.orchestra.run.smithi055.stdout:UBUNTU_CODENAME=jammy 2023-12-12T01:29:13.684 INFO:teuthology.lock.ops:Updating smithi055.front.sepia.ceph.com on lock server 2023-12-12T01:29:13.704 DEBUG:teuthology.orchestra.run.smithi140:> uname -m 2023-12-12T01:29:13.710 INFO:teuthology.orchestra.run.smithi140.stdout:x86_64 2023-12-12T01:29:13.710 DEBUG:teuthology.orchestra.run.smithi140:> cat /etc/os-release 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:NAME="Ubuntu" 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_ID="22.04" 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:VERSION_CODENAME=jammy 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:ID=ubuntu 2023-12-12T01:29:13.758 INFO:teuthology.orchestra.run.smithi140.stdout:ID_LIKE=debian 2023-12-12T01:29:13.759 INFO:teuthology.orchestra.run.smithi140.stdout:HOME_URL="https://www.ubuntu.com/" 2023-12-12T01:29:13.759 INFO:teuthology.orchestra.run.smithi140.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2023-12-12T01:29:13.759 INFO:teuthology.orchestra.run.smithi140.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2023-12-12T01:29:13.759 INFO:teuthology.orchestra.run.smithi140.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2023-12-12T01:29:13.759 INFO:teuthology.orchestra.run.smithi140.stdout:UBUNTU_CODENAME=jammy 2023-12-12T01:29:13.759 INFO:teuthology.lock.ops:Updating smithi140.front.sepia.ceph.com on lock server 2023-12-12T01:29:13.779 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-12T01:29:13.791 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-12T01:29:13.802 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-12T01:29:13.803 DEBUG:teuthology.orchestra.run.smithi055:> test '!' -e /home/ubuntu/cephtest 2023-12-12T01:29:13.805 DEBUG:teuthology.orchestra.run.smithi140:> test '!' -e /home/ubuntu/cephtest 2023-12-12T01:29:13.809 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-12T01:29:13.820 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-12T01:29:13.821 DEBUG:teuthology.orchestra.run.smithi055:> test -z $(ls -A /var/lib/ceph) 2023-12-12T01:29:13.851 DEBUG:teuthology.orchestra.run.smithi140:> test -z $(ls -A /var/lib/ceph) 2023-12-12T01:29:13.862 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-12T01:29:13.922 INFO:teuthology.run_tasks:Running task kernel... 2023-12-12T01:29:13.940 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-12T01:29:13.940 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-12T01:29:13.940 DEBUG:teuthology.orchestra.run.smithi055:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T01:29:13.941 DEBUG:teuthology.orchestra.run.smithi140:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-12T01:29:13.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:29:13.945 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-12-12T01:29:13.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:29:13.946 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2023-12-12T01:29:13.994 INFO:teuthology.orchestra.run.smithi140.stdout:5.15.0-56-generic 2023-12-12T01:29:13.994 INFO:teuthology.task.kernel:Running kernel on smithi140: 5.15.0-56-generic 2023-12-12T01:29:13.994 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get clean 2023-12-12T01:29:13.995 INFO:teuthology.orchestra.run.smithi055.stdout:5.15.0-56-generic 2023-12-12T01:29:13.995 INFO:teuthology.task.kernel:Running kernel on smithi055: 5.15.0-56-generic 2023-12-12T01:29:13.995 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get clean 2023-12-12T01:29:14.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2023-12-12T01:29:14.118 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get update 2023-12-12T01:29:14.264 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:29:14.264 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-12T01:29:14.264 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-12T01:29:14.294 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:29:14.294 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2023-12-12T01:29:14.294 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2023-12-12T01:29:14.330 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-12T01:29:14.380 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2023-12-12T01:29:14.501 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [379 kB] 2023-12-12T01:29:14.539 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,035 kB] 2023-12-12T01:29:14.563 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,035 kB] 2023-12-12T01:29:14.575 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [543 kB] 2023-12-12T01:29:14.595 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [197 kB] 2023-12-12T01:29:14.598 INFO:teuthology.orchestra.run.smithi140.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-12T01:29:14.599 INFO:teuthology.orchestra.run.smithi140.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-12T01:29:14.599 INFO:teuthology.orchestra.run.smithi140.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,205 kB] 2023-12-12T01:29:14.614 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [543 kB] 2023-12-12T01:29:14.618 INFO:teuthology.orchestra.run.smithi140.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [195 kB] 2023-12-12T01:29:14.620 INFO:teuthology.orchestra.run.smithi140.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-12T01:29:14.620 INFO:teuthology.orchestra.run.smithi140.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [577 kB] 2023-12-12T01:29:14.626 INFO:teuthology.orchestra.run.smithi140.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [815 kB] 2023-12-12T01:29:14.634 INFO:teuthology.orchestra.run.smithi140.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [152 kB] 2023-12-12T01:29:14.635 INFO:teuthology.orchestra.run.smithi140.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,246 kB] 2023-12-12T01:29:14.636 INFO:teuthology.orchestra.run.smithi140.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-12T01:29:14.637 INFO:teuthology.orchestra.run.smithi140.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-12T01:29:14.637 INFO:teuthology.orchestra.run.smithi140.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-12T01:29:14.637 INFO:teuthology.orchestra.run.smithi140.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-12T01:29:14.645 INFO:teuthology.orchestra.run.smithi140.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-12T01:29:14.651 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [379 kB] 2023-12-12T01:29:14.658 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [197 kB] 2023-12-12T01:29:14.664 INFO:teuthology.orchestra.run.smithi140.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [257 kB] 2023-12-12T01:29:14.666 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2023-12-12T01:29:14.666 INFO:teuthology.orchestra.run.smithi055.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,205 kB] 2023-12-12T01:29:14.669 INFO:teuthology.orchestra.run.smithi140.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-12T01:29:14.669 INFO:teuthology.orchestra.run.smithi140.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-12T01:29:14.669 INFO:teuthology.orchestra.run.smithi140.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,226 kB] 2023-12-12T01:29:14.680 INFO:teuthology.orchestra.run.smithi140.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [199 kB] 2023-12-12T01:29:14.682 INFO:teuthology.orchestra.run.smithi140.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-12T01:29:14.682 INFO:teuthology.orchestra.run.smithi140.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,018 kB] 2023-12-12T01:29:14.687 INFO:teuthology.orchestra.run.smithi055.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,246 kB] 2023-12-12T01:29:14.687 INFO:teuthology.orchestra.run.smithi055.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [32.4 kB] 2023-12-12T01:29:14.688 INFO:teuthology.orchestra.run.smithi055.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [195 kB] 2023-12-12T01:29:14.689 INFO:teuthology.orchestra.run.smithi140.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-12T01:29:14.691 INFO:teuthology.orchestra.run.smithi055.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2023-12-12T01:29:14.691 INFO:teuthology.orchestra.run.smithi055.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [815 kB] 2023-12-12T01:29:14.694 INFO:teuthology.orchestra.run.smithi140.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-12T01:29:14.696 INFO:teuthology.orchestra.run.smithi140.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-12T01:29:14.699 INFO:teuthology.orchestra.run.smithi140.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-12T01:29:14.699 INFO:teuthology.orchestra.run.smithi140.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-12T01:29:14.700 INFO:teuthology.orchestra.run.smithi140.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-12T01:29:14.700 INFO:teuthology.orchestra.run.smithi140.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-12T01:29:14.703 INFO:teuthology.orchestra.run.smithi055.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [577 kB] 2023-12-12T01:29:14.710 INFO:teuthology.orchestra.run.smithi055.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [152 kB] 2023-12-12T01:29:14.713 INFO:teuthology.orchestra.run.smithi140.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-12T01:29:14.714 INFO:teuthology.orchestra.run.smithi140.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-12T01:29:14.714 INFO:teuthology.orchestra.run.smithi140.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-12T01:29:14.715 INFO:teuthology.orchestra.run.smithi140.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-12T01:29:14.715 INFO:teuthology.orchestra.run.smithi140.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-12T01:29:14.715 INFO:teuthology.orchestra.run.smithi140.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-12T01:29:14.732 INFO:teuthology.orchestra.run.smithi140.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-12T01:29:14.733 INFO:teuthology.orchestra.run.smithi140.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-12T01:29:14.764 INFO:teuthology.orchestra.run.smithi055.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [257 kB] 2023-12-12T01:29:14.764 INFO:teuthology.orchestra.run.smithi055.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2023-12-12T01:29:14.764 INFO:teuthology.orchestra.run.smithi055.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,032 B] 2023-12-12T01:29:14.764 INFO:teuthology.orchestra.run.smithi055.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [36.5 kB] 2023-12-12T01:29:14.765 INFO:teuthology.orchestra.run.smithi055.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,060 B] 2023-12-12T01:29:14.765 INFO:teuthology.orchestra.run.smithi055.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2023-12-12T01:29:14.767 INFO:teuthology.orchestra.run.smithi055.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2023-12-12T01:29:14.767 INFO:teuthology.orchestra.run.smithi055.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,226 kB] 2023-12-12T01:29:14.780 INFO:teuthology.orchestra.run.smithi055.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [32.8 kB] 2023-12-12T01:29:14.780 INFO:teuthology.orchestra.run.smithi055.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [199 kB] 2023-12-12T01:29:14.782 INFO:teuthology.orchestra.run.smithi055.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2023-12-12T01:29:14.783 INFO:teuthology.orchestra.run.smithi055.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,018 kB] 2023-12-12T01:29:14.795 INFO:teuthology.orchestra.run.smithi055.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [675 kB] 2023-12-12T01:29:14.801 INFO:teuthology.orchestra.run.smithi055.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [226 kB] 2023-12-12T01:29:14.803 INFO:teuthology.orchestra.run.smithi055.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2023-12-12T01:29:14.804 INFO:teuthology.orchestra.run.smithi055.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [3,888 B] 2023-12-12T01:29:14.804 INFO:teuthology.orchestra.run.smithi055.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [41.6 kB] 2023-12-12T01:29:14.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [9,768 B] 2023-12-12T01:29:14.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2023-12-12T01:29:14.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [64.2 kB] 2023-12-12T01:29:14.807 INFO:teuthology.orchestra.run.smithi055.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [56.3 kB] 2023-12-12T01:29:14.825 INFO:teuthology.orchestra.run.smithi055.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [10.5 kB] 2023-12-12T01:29:14.825 INFO:teuthology.orchestra.run.smithi055.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2023-12-12T01:29:14.826 INFO:teuthology.orchestra.run.smithi055.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.8 kB] 2023-12-12T01:29:14.826 INFO:teuthology.orchestra.run.smithi055.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.8 kB] 2023-12-12T01:29:14.826 INFO:teuthology.orchestra.run.smithi055.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2023-12-12T01:29:14.826 INFO:teuthology.orchestra.run.smithi055.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2023-12-12T01:29:19.964 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 10.7 MB in 2s (6,975 kB/s) 2023-12-12T01:29:20.312 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 10.7 MB in 2s (6,451 kB/s) 2023-12-12T01:29:21.268 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:29:21.287 DEBUG:teuthology.orchestra.run.smithi140:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-12T01:29:21.355 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:29:21.541 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2023-12-12T01:29:21.541 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2023-12-12T01:29:21.704 INFO:teuthology.orchestra.run.smithi140.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:29:21.704 INFO:teuthology.orchestra.run.smithi140.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:29:21.704 INFO:teuthology.orchestra.run.smithi140.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:29:21.707 INFO:teuthology.orchestra.run.smithi140.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:29:21.707 INFO:teuthology.orchestra.run.smithi140.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-12T01:29:21.707 INFO:teuthology.orchestra.run.smithi140.stdout: python2.7-minimal 2023-12-12T01:29:21.707 INFO:teuthology.orchestra.run.smithi140.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:29:21.708 INFO:teuthology.orchestra.run.smithi140.stdout:The following additional packages will be installed: 2023-12-12T01:29:21.708 INFO:teuthology.orchestra.run.smithi140.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-12T01:29:21.708 INFO:teuthology.orchestra.run.smithi140.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-12T01:29:21.709 INFO:teuthology.orchestra.run.smithi140.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-12T01:29:21.709 INFO:teuthology.orchestra.run.smithi140.stdout:Suggested packages: 2023-12-12T01:29:21.709 INFO:teuthology.orchestra.run.smithi140.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-12T01:29:21.709 INFO:teuthology.orchestra.run.smithi140.stdout:Recommended packages: 2023-12-12T01:29:21.709 INFO:teuthology.orchestra.run.smithi140.stdout: thermald 2023-12-12T01:29:21.758 INFO:teuthology.orchestra.run.smithi140.stdout:The following NEW packages will be installed: 2023-12-12T01:29:21.758 INFO:teuthology.orchestra.run.smithi140.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-12T01:29:21.758 INFO:teuthology.orchestra.run.smithi140.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-12T01:29:21.768 INFO:teuthology.orchestra.run.smithi140.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-12T01:29:21.768 INFO:teuthology.orchestra.run.smithi140.stdout:The following packages will be upgraded: 2023-12-12T01:29:21.768 INFO:teuthology.orchestra.run.smithi140.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-12T01:29:21.809 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:29:21.816 INFO:teuthology.orchestra.run.smithi140.stdout:3 upgraded, 5 newly installed, 0 to remove and 275 not upgraded. 2023-12-12T01:29:21.816 INFO:teuthology.orchestra.run.smithi140.stdout:Need to get 113 MB of archives. 2023-12-12T01:29:21.816 INFO:teuthology.orchestra.run.smithi140.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-12T01:29:21.816 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-12T01:29:21.829 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-12T01:29:21.895 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:29:22.090 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2023-12-12T01:29:22.090 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2023-12-12T01:29:22.259 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:29:22.259 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:29:22.259 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:29:22.260 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:29:22.260 INFO:teuthology.orchestra.run.smithi055.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2023-12-12T01:29:22.260 INFO:teuthology.orchestra.run.smithi055.stdout: python2.7-minimal 2023-12-12T01:29:22.260 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:29:22.261 INFO:teuthology.orchestra.run.smithi055.stdout:The following additional packages will be installed: 2023-12-12T01:29:22.261 INFO:teuthology.orchestra.run.smithi055.stdout: linux-generic linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-12T01:29:22.261 INFO:teuthology.orchestra.run.smithi055.stdout: linux-headers-generic linux-image-5.15.0-91-generic 2023-12-12T01:29:22.262 INFO:teuthology.orchestra.run.smithi055.stdout: linux-modules-5.15.0-91-generic linux-modules-extra-5.15.0-91-generic 2023-12-12T01:29:22.263 INFO:teuthology.orchestra.run.smithi055.stdout:Suggested packages: 2023-12-12T01:29:22.263 INFO:teuthology.orchestra.run.smithi055.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2023-12-12T01:29:22.263 INFO:teuthology.orchestra.run.smithi055.stdout:Recommended packages: 2023-12-12T01:29:22.263 INFO:teuthology.orchestra.run.smithi055.stdout: thermald 2023-12-12T01:29:22.293 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-12T01:29:22.320 INFO:teuthology.orchestra.run.smithi055.stdout:The following NEW packages will be installed: 2023-12-12T01:29:22.321 INFO:teuthology.orchestra.run.smithi055.stdout: linux-headers-5.15.0-91 linux-headers-5.15.0-91-generic 2023-12-12T01:29:22.321 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.15.0-91-generic linux-modules-5.15.0-91-generic 2023-12-12T01:29:22.321 INFO:teuthology.orchestra.run.smithi055.stdout: linux-modules-extra-5.15.0-91-generic 2023-12-12T01:29:22.322 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages will be upgraded: 2023-12-12T01:29:22.323 INFO:teuthology.orchestra.run.smithi055.stdout: linux-generic linux-headers-generic linux-image-generic 2023-12-12T01:29:22.387 INFO:teuthology.orchestra.run.smithi055.stdout:3 upgraded, 5 newly installed, 0 to remove and 275 not upgraded. 2023-12-12T01:29:22.387 INFO:teuthology.orchestra.run.smithi055.stdout:Need to get 113 MB of archives. 2023-12-12T01:29:22.388 INFO:teuthology.orchestra.run.smithi055.stdout:After this operation, 582 MB of additional disk space will be used. 2023-12-12T01:29:22.388 INFO:teuthology.orchestra.run.smithi055.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-91-generic amd64 5.15.0-91.101 [22.5 MB] 2023-12-12T01:29:22.418 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-12T01:29:23.170 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-91-generic amd64 5.15.0-91.101 [11.5 MB] 2023-12-12T01:29:23.391 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-91-generic amd64 5.15.0-91.101 [63.8 MB] 2023-12-12T01:29:23.735 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-12T01:29:23.735 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-12T01:29:23.735 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-12T01:29:23.981 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-12T01:29:24.043 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-12T01:29:24.446 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 113 MB in 2s (49.6 MB/s) 2023-12-12T01:29:24.527 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.91.88 [1,700 B] 2023-12-12T01:29:24.527 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.91.88 [2,488 B] 2023-12-12T01:29:24.527 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91 all 5.15.0-91.101 [12.3 MB] 2023-12-12T01:29:24.603 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-12T01:29:24.757 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-91-generic amd64 5.15.0-91.101 [2,860 kB] 2023-12-12T01:29:24.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.91.88 [2,342 B] 2023-12-12T01:29:25.211 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 113 MB in 2s (45.7 MB/s) 2023-12-12T01:29:25.375 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-modules-5.15.0-91-generic. 2023-12-12T01:29:26.580 INFO:teuthology.orchestra.run.smithi140.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-12T01:29:26.585 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:26.650 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:27.455 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2023-12-12T01:29:27.460 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../0-linux-modules-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:27.572 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:28.809 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-12T01:29:28.831 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:29.179 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:29.612 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-12T01:29:29.633 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:29.670 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:29.739 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-image-5.15.0-91-generic. 2023-12-12T01:29:29.761 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../1-linux-image-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:29.850 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:30.301 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-91-generic. 2023-12-12T01:29:30.322 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:30.376 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:35.503 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:35.621 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:35.929 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:36.050 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:36.328 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-12T01:29:36.339 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-12T01:29:36.377 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-12T01:29:37.052 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../3-linux-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:37.177 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:37.552 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:37.680 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-image-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:38.025 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-headers-5.15.0-91. 2023-12-12T01:29:38.048 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../5-linux-headers-5.15.0-91_5.15.0-91.101_all.deb ... 2023-12-12T01:29:38.108 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-12T01:29:41.899 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-12T01:29:41.937 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:41.974 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:44.291 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package linux-headers-5.15.0-91-generic. 2023-12-12T01:29:44.329 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../6-linux-headers-5.15.0-91-generic_5.15.0-91.101_amd64.deb ... 2023-12-12T01:29:44.366 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:44.485 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:44.770 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:45.094 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-12T01:29:45.216 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:45.342 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-12T01:29:45.471 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:46.798 INFO:teuthology.orchestra.run.smithi140.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:29:46.798 INFO:teuthology.orchestra.run.smithi140.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-12T01:29:46.798 INFO:teuthology.orchestra.run.smithi140.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:29:46.798 INFO:teuthology.orchestra.run.smithi140.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-12T01:29:46.956 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:47.225 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.91.88_amd64.deb ... 2023-12-12T01:29:47.343 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking linux-headers-generic (5.15.0.91.88) over (5.15.0.56.54) ... 2023-12-12T01:29:47.719 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-5.15.0-91 (5.15.0-91.101) ... 2023-12-12T01:29:47.831 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:47.966 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-headers-generic (5.15.0.91.88) ... 2023-12-12T01:29:48.086 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:48.350 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-12T01:29:48.475 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:49.395 INFO:teuthology.orchestra.run.smithi055.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:29:49.395 INFO:teuthology.orchestra.run.smithi055.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2023-12-12T01:29:49.395 INFO:teuthology.orchestra.run.smithi055.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:29:49.395 INFO:teuthology.orchestra.run.smithi055.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-91-generic 2023-12-12T01:29:49.563 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-modules-extra-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:50.353 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-12T01:29:50.462 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:50.528 INFO:teuthology.orchestra.run.smithi140.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-12T01:29:50.529 INFO:teuthology.orchestra.run.smithi140.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:29:50.998 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-image-generic (5.15.0.91.88) ... 2023-12-12T01:29:51.123 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-modules-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:52.542 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up linux-generic (5.15.0.91.88) ... 2023-12-12T01:29:52.660 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for linux-image-5.15.0-91-generic (5.15.0-91.101) ... 2023-12-12T01:29:52.868 INFO:teuthology.orchestra.run.smithi055.stdout:/etc/kernel/postinst.d/initramfs-tools: 2023-12-12T01:29:52.868 INFO:teuthology.orchestra.run.smithi055.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:30:04.801 INFO:teuthology.orchestra.run.smithi140.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-12T01:30:04.801 INFO:teuthology.orchestra.run.smithi140.stdout:Sourcing file `/etc/default/grub' 2023-12-12T01:30:04.808 INFO:teuthology.orchestra.run.smithi140.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-12T01:30:04.831 INFO:teuthology.orchestra.run.smithi140.stdout:Generating grub configuration file ... 2023-12-12T01:30:05.097 INFO:teuthology.orchestra.run.smithi140.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:30:05.111 INFO:teuthology.orchestra.run.smithi140.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:30:05.419 INFO:teuthology.orchestra.run.smithi140.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:30:05.424 INFO:teuthology.orchestra.run.smithi140.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-12T01:30:05.492 INFO:teuthology.orchestra.run.smithi140.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-12T01:30:05.497 INFO:teuthology.orchestra.run.smithi140.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-12T01:30:05.649 INFO:teuthology.orchestra.run.smithi140.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-12T01:30:05.649 INFO:teuthology.orchestra.run.smithi140.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-12T01:30:05.649 INFO:teuthology.orchestra.run.smithi140.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-12T01:30:05.693 INFO:teuthology.orchestra.run.smithi140.stdout:done 2023-12-12T01:30:06.122 DEBUG:teuthology.orchestra.run.smithi140:> dpkg -s linux-image-generic 2023-12-12T01:30:06.143 INFO:teuthology.orchestra.run.smithi140.stdout:Package: linux-image-generic 2023-12-12T01:30:06.143 INFO:teuthology.orchestra.run.smithi140.stdout:Status: install ok installed 2023-12-12T01:30:06.143 INFO:teuthology.orchestra.run.smithi140.stdout:Priority: optional 2023-12-12T01:30:06.143 INFO:teuthology.orchestra.run.smithi140.stdout:Section: kernel 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Installed-Size: 20 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Maintainer: Ubuntu Kernel Team 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Architecture: amd64 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Source: linux-meta 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Version: 5.15.0.91.88 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Recommends: thermald 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout:Description: Generic Linux kernel image 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout: This package will always depend on the latest generic kernel image 2023-12-12T01:30:06.144 INFO:teuthology.orchestra.run.smithi140.stdout: available. 2023-12-12T01:30:06.144 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-12T01:30:06.145 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T01:30:06.145 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T01:30:06.145 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2023-12-12T01:30:06.145 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi140.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T01:30:06.145 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get clean 2023-12-12T01:30:06.307 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get update 2023-12-12T01:30:06.431 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-12T01:30:06.549 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:30:06.629 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-12T01:30:06.726 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-12T01:30:07.701 INFO:teuthology.orchestra.run.smithi055.stdout:/etc/kernel/postinst.d/zz-update-grub: 2023-12-12T01:30:07.702 INFO:teuthology.orchestra.run.smithi055.stdout:Sourcing file `/etc/default/grub' 2023-12-12T01:30:07.729 INFO:teuthology.orchestra.run.smithi055.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-12T01:30:07.750 INFO:teuthology.orchestra.run.smithi055.stdout:Generating grub configuration file ... 2023-12-12T01:30:07.788 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:30:07.807 DEBUG:teuthology.orchestra.run.smithi140:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-12T01:30:07.872 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:30:08.045 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:30:08.062 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2023-12-12T01:30:08.062 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2023-12-12T01:30:08.067 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:30:08.232 INFO:teuthology.orchestra.run.smithi140.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-12T01:30:08.232 INFO:teuthology.orchestra.run.smithi140.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-12T01:30:08.233 INFO:teuthology.orchestra.run.smithi140.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:30:08.275 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 0 newly installed, 0 to remove and 275 not upgraded. 2023-12-12T01:30:08.277 DEBUG:teuthology.orchestra.run.smithi140:> dpkg -s linux-image-generic 2023-12-12T01:30:08.292 INFO:teuthology.orchestra.run.smithi140.stdout:Package: linux-image-generic 2023-12-12T01:30:08.292 INFO:teuthology.orchestra.run.smithi140.stdout:Status: install ok installed 2023-12-12T01:30:08.292 INFO:teuthology.orchestra.run.smithi140.stdout:Priority: optional 2023-12-12T01:30:08.292 INFO:teuthology.orchestra.run.smithi140.stdout:Section: kernel 2023-12-12T01:30:08.292 INFO:teuthology.orchestra.run.smithi140.stdout:Installed-Size: 20 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Maintainer: Ubuntu Kernel Team 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Architecture: amd64 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Source: linux-meta 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Version: 5.15.0.91.88 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Recommends: thermald 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout:Description: Generic Linux kernel image 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout: This package will always depend on the latest generic kernel image 2023-12-12T01:30:08.293 INFO:teuthology.orchestra.run.smithi140.stdout: available. 2023-12-12T01:30:08.294 DEBUG:teuthology.orchestra.run.smithi140:> mktemp 2023-12-12T01:30:08.341 INFO:teuthology.orchestra.run.smithi140.stdout:/tmp/tmp.GSEeC3RXsN 2023-12-12T01:30:08.341 DEBUG:teuthology.orchestra.run.smithi140:> sudo cp /boot/grub/grub.cfg /tmp/tmp.GSEeC3RXsN 2023-12-12T01:30:08.397 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0666 /tmp/tmp.GSEeC3RXsN 2023-12-12T01:30:08.410 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:30:08.416 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-12T01:30:08.487 INFO:teuthology.orchestra.run.smithi055.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-12T01:30:08.493 INFO:teuthology.orchestra.run.smithi055.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-12T01:30:08.518 DEBUG:teuthology.orchestra.remote:smithi140:/tmp/tmp.GSEeC3RXsN is 10KB 2023-12-12T01:30:08.531 DEBUG:teuthology.orchestra.run.smithi140:> rm -fr /tmp/tmp.GSEeC3RXsN 2023-12-12T01:30:08.537 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-12T01:30:08.593 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:30:08.593 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-12T01:30:08.593 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-12T01:30:08.657 INFO:teuthology.orchestra.run.smithi055.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-12T01:30:08.657 INFO:teuthology.orchestra.run.smithi055.stdout:Systems on them will not be added to the GRUB boot configuration. 2023-12-12T01:30:08.657 INFO:teuthology.orchestra.run.smithi055.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-12T01:30:08.660 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-12T01:30:08.660 DEBUG:teuthology.orchestra.run.smithi140:> sudo update-grub 2023-12-12T01:30:08.696 INFO:teuthology.orchestra.run.smithi055.stdout:done 2023-12-12T01:30:09.189 DEBUG:teuthology.orchestra.run.smithi055:> dpkg -s linux-image-generic 2023-12-12T01:30:09.213 INFO:teuthology.orchestra.run.smithi055.stdout:Package: linux-image-generic 2023-12-12T01:30:09.213 INFO:teuthology.orchestra.run.smithi055.stdout:Status: install ok installed 2023-12-12T01:30:09.213 INFO:teuthology.orchestra.run.smithi055.stdout:Priority: optional 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Section: kernel 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Installed-Size: 20 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Maintainer: Ubuntu Kernel Team 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: amd64 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Source: linux-meta 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Version: 5.15.0.91.88 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Recommends: thermald 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout:Description: Generic Linux kernel image 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout: This package will always depend on the latest generic kernel image 2023-12-12T01:30:09.214 INFO:teuthology.orchestra.run.smithi055.stdout: available. 2023-12-12T01:30:09.215 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-91-generic 2023-12-12T01:30:09.215 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-12T01:30:09.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-12T01:30:09.215 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2023-12-12T01:30:09.216 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi055.front.sepia.ceph.com, path=None, version=distro) 2023-12-12T01:30:09.216 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get clean 2023-12-12T01:30:09.426 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2023-12-12T01:30:09.551 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-12T01:30:09.754 INFO:teuthology.orchestra.run.smithi140.stderr:Sourcing file `/etc/default/grub' 2023-12-12T01:30:09.755 INFO:teuthology.orchestra.run.smithi140.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-12T01:30:09.758 INFO:teuthology.orchestra.run.smithi140.stderr:Generating grub configuration file ... 2023-12-12T01:30:09.927 INFO:teuthology.orchestra.run.smithi140.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:30:09.933 INFO:teuthology.orchestra.run.smithi140.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:30:10.048 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:30:10.067 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-12T01:30:10.086 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-12T01:30:10.183 INFO:teuthology.orchestra.run.smithi140.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:30:10.189 INFO:teuthology.orchestra.run.smithi140.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-12T01:30:10.229 INFO:teuthology.orchestra.run.smithi140.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-12T01:30:10.234 INFO:teuthology.orchestra.run.smithi140.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-12T01:30:10.359 INFO:teuthology.orchestra.run.smithi140.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-12T01:30:10.360 INFO:teuthology.orchestra.run.smithi140.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-12T01:30:10.360 INFO:teuthology.orchestra.run.smithi140.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-12T01:30:10.374 INFO:teuthology.orchestra.run.smithi140.stderr:done 2023-12-12T01:30:10.383 DEBUG:teuthology.orchestra.run.smithi140:> sudo shutdown -r now 2023-12-12T01:30:11.368 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:30:11.384 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2023-12-12T01:30:11.446 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:30:11.637 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2023-12-12T01:30:11.638 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2023-12-12T01:30:11.809 INFO:teuthology.orchestra.run.smithi055.stdout:linux-image-generic is already the newest version (5.15.0.91.88). 2023-12-12T01:30:11.809 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:30:11.810 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:30:11.810 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:30:11.810 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:30:11.810 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-12T01:30:11.810 INFO:teuthology.orchestra.run.smithi055.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-12T01:30:11.811 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:30:11.851 INFO:teuthology.orchestra.run.smithi055.stdout:0 upgraded, 0 newly installed, 0 to remove and 275 not upgraded. 2023-12-12T01:30:11.853 DEBUG:teuthology.orchestra.run.smithi055:> dpkg -s linux-image-generic 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Package: linux-image-generic 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Status: install ok installed 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Priority: optional 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Section: kernel 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Installed-Size: 20 2023-12-12T01:30:11.869 INFO:teuthology.orchestra.run.smithi055.stdout:Maintainer: Ubuntu Kernel Team 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Architecture: amd64 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Source: linux-meta 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Version: 5.15.0.91.88 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.1), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-91), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.1) 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Depends: linux-image-5.15.0-91-generic, linux-modules-extra-5.15.0-91-generic, linux-firmware, intel-microcode, amd64-microcode 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Recommends: thermald 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout:Description: Generic Linux kernel image 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout: This package will always depend on the latest generic kernel image 2023-12-12T01:30:11.870 INFO:teuthology.orchestra.run.smithi055.stdout: available. 2023-12-12T01:30:11.871 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2023-12-12T01:30:11.917 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.cWwM3UWero 2023-12-12T01:30:11.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo cp /boot/grub/grub.cfg /tmp/tmp.cWwM3UWero 2023-12-12T01:30:11.972 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.cWwM3UWero 2023-12-12T01:30:12.086 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.cWwM3UWero is 10KB 2023-12-12T01:30:12.098 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.cWwM3UWero 2023-12-12T01:30:12.104 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2023-12-12T01:30:12.162 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:30:12.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/grub.d/01_ceph_kernel 2023-12-12T01:30:12.162 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2023-12-12T01:30:12.255 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-91-generic 2023-12-12T01:30:12.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo update-grub 2023-12-12T01:30:13.587 INFO:teuthology.orchestra.run.smithi055.stderr:Sourcing file `/etc/default/grub' 2023-12-12T01:30:13.588 INFO:teuthology.orchestra.run.smithi055.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2023-12-12T01:30:13.591 INFO:teuthology.orchestra.run.smithi055.stderr:Generating grub configuration file ... 2023-12-12T01:30:13.771 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.15.0-91-generic 2023-12-12T01:30:13.777 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.15.0-91-generic 2023-12-12T01:30:14.038 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2023-12-12T01:30:14.044 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2023-12-12T01:30:14.087 INFO:teuthology.orchestra.run.smithi055.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2023-12-12T01:30:14.092 INFO:teuthology.orchestra.run.smithi055.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2023-12-12T01:30:14.232 INFO:teuthology.orchestra.run.smithi055.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2023-12-12T01:30:14.232 INFO:teuthology.orchestra.run.smithi055.stderr:Systems on them will not be added to the GRUB boot configuration. 2023-12-12T01:30:14.232 INFO:teuthology.orchestra.run.smithi055.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2023-12-12T01:30:14.256 INFO:teuthology.orchestra.run.smithi055.stderr:done 2023-12-12T01:30:14.265 DEBUG:teuthology.orchestra.run.smithi055:> sudo shutdown -r now 2023-12-12T01:30:40.410 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2023-12-12T01:30:40.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:30:40.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:30:44.270 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2023-12-12T01:30:44.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:30:44.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:30:58.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-12T01:31:07.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:31:07.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:31:10.854 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-12T01:31:22.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:31:22.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:31:25.926 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.140 2023-12-12T01:31:40.939 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:31:40.940 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:31:44.276 DEBUG:teuthology.orchestra.remote:timed out 2023-12-12T01:31:53.282 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:31:53.283 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:31:53.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.55 2023-12-12T01:32:05.294 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:32:05.295 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:32:05.502 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-12-12T01:32:06.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:32:06.120 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-91-generic"... 2023-12-12T01:32:06.120 DEBUG:teuthology.orchestra.run.smithi055:> uname -r 2023-12-12T01:32:06.167 INFO:teuthology.orchestra.run.smithi055.stdout:5.15.0-91-generic 2023-12-12T01:32:06.167 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-12T01:32:06.167 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T01:32:06.168 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-12T01:32:07.169 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2023-12-12T01:32:07.169 DEBUG:teuthology.orchestra.run.smithi055:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T01:32:07.239 INFO:teuthology.orchestra.run.smithi055.stdout:ttyS1 2023-12-12T01:32:07.257 DEBUG:teuthology.parallel:result is None 2023-12-12T01:32:12.798 DEBUG:teuthology.orchestra.run.smithi140:> true 2023-12-12T01:32:13.368 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:32:13.368 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-91-generic"... 2023-12-12T01:32:13.368 DEBUG:teuthology.orchestra.run.smithi140:> uname -r 2023-12-12T01:32:13.417 INFO:teuthology.orchestra.run.smithi140.stdout:5.15.0-91-generic 2023-12-12T01:32:13.418 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-91-generic vs 5.15.0-91-generic 2023-12-12T01:32:13.418 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-12T01:32:13.418 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2023-12-12T01:32:14.419 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2023-12-12T01:32:14.419 DEBUG:teuthology.orchestra.run.smithi140:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-12T01:32:14.747 INFO:teuthology.orchestra.run.smithi140.stdout:ttyS1 2023-12-12T01:32:14.779 DEBUG:teuthology.parallel:result is None 2023-12-12T01:32:14.780 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-12T01:32:14.788 INFO:teuthology.task.internal:Creating test directory... 2023-12-12T01:32:14.789 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T01:32:14.791 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-12T01:32:14.797 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-12T01:32:14.809 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-12T01:32:14.816 INFO:teuthology.task.internal:Creating archive directory... 2023-12-12T01:32:14.816 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T01:32:14.840 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-12T01:32:14.868 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-12T01:32:14.875 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-12T01:32:14.876 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T01:32:14.904 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-12T01:32:14.918 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:32:14.927 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:32:14.944 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:32:14.955 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-12T01:32:14.957 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-12T01:32:14.965 INFO:teuthology.task.internal:Configuring sudo... 2023-12-12T01:32:14.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T01:32:14.972 DEBUG:teuthology.orchestra.run.smithi140:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-12T01:32:15.013 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-12T01:32:15.022 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-12T01:32:15.022 DEBUG:teuthology.orchestra.run.smithi055:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T01:32:15.028 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-12T01:32:15.062 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:32:15.079 DEBUG:teuthology.orchestra.run.smithi055:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:32:15.127 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:32:15.127 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T01:32:15.188 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-12T01:32:15.194 DEBUG:teuthology.orchestra.run.smithi140:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-12T01:32:15.241 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:32:15.242 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-12T01:32:15.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo service rsyslog restart 2023-12-12T01:32:15.311 DEBUG:teuthology.orchestra.run.smithi140:> sudo service rsyslog restart 2023-12-12T01:32:15.379 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-12T01:32:15.388 INFO:teuthology.task.internal:Starting timer... 2023-12-12T01:32:15.389 INFO:teuthology.run_tasks:Running task pcp... 2023-12-12T01:32:15.398 INFO:teuthology.run_tasks:Running task selinux... 2023-12-12T01:32:15.406 DEBUG:teuthology.task.selinux:Excluding smithi055: OS 'ubuntu' does not support SELinux 2023-12-12T01:32:15.407 DEBUG:teuthology.task.selinux:Excluding smithi140: OS 'ubuntu' does not support SELinux 2023-12-12T01:32:15.407 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-12T01:32:15.407 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2023-12-12T01:32:15.407 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-12T01:32:15.407 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-12T01:32:15.416 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-12-12T01:32:15.422 INFO:teuthology.repo_utils:Fetching main from origin 2023-12-12T01:32:15.545 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-12T01:32:15.559 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-12T01:32:15.560 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi055.front.sepia.ceph.com,smithi140.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-12T01:40:11.687 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi055.front.sepia.ceph.com'), Remote(name='ubuntu@smithi140.front.sepia.ceph.com')] 2023-12-12T01:40:11.688 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:40:11.689 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi055.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:40:11.774 DEBUG:teuthology.orchestra.run.smithi055:> true 2023-12-12T01:40:11.850 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi055.front.sepia.ceph.com' 2023-12-12T01:40:11.851 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:40:11.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi140.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-12T01:40:11.927 DEBUG:teuthology.orchestra.run.smithi140:> true 2023-12-12T01:40:12.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi140.front.sepia.ceph.com' 2023-12-12T01:40:12.003 INFO:teuthology.run_tasks:Running task clock... 2023-12-12T01:40:12.013 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-12T01:40:12.013 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-12T01:40:12.014 DEBUG:teuthology.orchestra.run.smithi055:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-12T01:40:12.016 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-12T01:40:12.016 DEBUG:teuthology.orchestra.run.smithi140:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-12T01:40:12.044 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Command line: ntpd -gq 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: ---------------------------------------------------- 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: ntp-4 is maintained by Network Time Foundation, 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: corporation. Support and training for ntp-4 are 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: available at https://www.nwtime.org/support 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: ---------------------------------------------------- 2023-12-12T01:40:12.045 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: proto: precision = 0.066 usec (-24) 2023-12-12T01:40:12.046 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: basedate set to 2022-02-04 2023-12-12T01:40:12.046 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: gps base set to 2022-02-06 (week 2196) 2023-12-12T01:40:12.046 INFO:teuthology.orchestra.run.smithi055.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-12T01:40:12.046 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-12T01:40:12.046 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: restrict ::: KOD does nothing without LIMITED. 2023-12-12T01:40:12.047 INFO:teuthology.orchestra.run.smithi055.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-12T01:40:12.047 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen and drop on 0 v6wildcard [::]:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 2 lo 127.0.0.1:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 3 ens1f0 172.21.15.55:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 4 lo [::1]:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2606%4]:123 2023-12-12T01:40:12.048 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:12 ntpd[27891]: Listening on routing socket on fd #22 for interface updates 2023-12-12T01:40:12.074 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Command line: ntpd -gq 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: ---------------------------------------------------- 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: ntp-4 is maintained by Network Time Foundation, 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: corporation. Support and training for ntp-4 are 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: available at https://www.nwtime.org/support 2023-12-12T01:40:12.075 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: ---------------------------------------------------- 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: proto: precision = 0.045 usec (-24) 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: basedate set to 2022-02-04 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: gps base set to 2022-02-06 (week 2196) 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2023-12-12T01:40:12.076 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: restrict ::: KOD does nothing without LIMITED. 2023-12-12T01:40:12.077 INFO:teuthology.orchestra.run.smithi140.stderr:restrict ::: KOD does nothing without LIMITED. 2023-12-12T01:40:12.077 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen and drop on 0 v6wildcard [::]:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 2 lo 127.0.0.1:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 3 ens1f1 172.21.15.140:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 4 lo [::1]:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listen normally on 5 ens1f1 [fe80::ae1f:6bff:fea5:ca31%5]:123 2023-12-12T01:40:12.078 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:12 ntpd[27891]: Listening on routing socket on fd #22 for interface updates 2023-12-12T01:40:13.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:13.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:13.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:13 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:14.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:14.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:14.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:14.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:14.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:14.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:14.075 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:14.075 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:14.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:14.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:14.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:14.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:14 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:15.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:15.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:15.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:15.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:15 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:16.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:16.075 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:16.075 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:16.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:16.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:16.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:16.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:16 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:17.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:17.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:17.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:17 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:18.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:18.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:18.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:18.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:18.045 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:18.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:18.075 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:18.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:18.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:18.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:18.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:18.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:18 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:19.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.046 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:19.047 INFO:teuthology.orchestra.run.smithi055.stdout:12 Dec 01:40:19 ntpd[27891]: ntpd: time slew -0.000229 s 2023-12-12T01:40:19.048 INFO:teuthology.orchestra.run.smithi055.stdout:ntpd: time slew -0.000229s 2023-12-12T01:40:19.048 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-12T01:40:19.048 INFO:teuthology.orchestra.run.smithi055.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/loopstats.20231212: Permission denied 2023-12-12T01:40:19.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.076 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/rawstats.20231212: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/peerstats.20231212: Permission denied 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stdout:12 Dec 01:40:19 ntpd[27891]: ntpd: time slew -0.000663 s 2023-12-12T01:40:19.077 INFO:teuthology.orchestra.run.smithi140.stdout:ntpd: time slew -0.000663s 2023-12-12T01:40:19.078 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2023-12-12T01:40:19.078 INFO:teuthology.orchestra.run.smithi140.stderr:12 Dec 01:40:19 ntpd[27891]: can't open /var/log/ntpstats/loopstats.20231212: Permission denied 2023-12-12T01:40:19.100 INFO:teuthology.orchestra.run.smithi055.stdout: remote refid st t when poll reach delay offset jitter 2023-12-12T01:40:19.100 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================== 2023-12-12T01:40:19.100 INFO:teuthology.orchestra.run.smithi055.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.100 INFO:teuthology.orchestra.run.smithi055.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.100 INFO:teuthology.orchestra.run.smithi055.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.101 INFO:teuthology.orchestra.run.smithi055.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout: remote refid st t when poll reach delay offset jitter 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout:============================================================================== 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.126 INFO:teuthology.orchestra.run.smithi140.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2023-12-12T01:40:19.127 INFO:teuthology.run_tasks:Running task install... 2023-12-12T01:40:19.136 DEBUG:teuthology.task.install:project ceph 2023-12-12T01:40:19.137 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}} 2023-12-12T01:40:19.137 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'} 2023-12-12T01:40:19.137 INFO:teuthology.task.install:Using flavor: default 2023-12-12T01:40:19.151 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-12T01:40:19.152 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2023-12-12T01:40:19.152 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-key list | grep Ceph 2023-12-12T01:40:19.153 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-key list | grep Ceph 2023-12-12T01:40:19.214 INFO:teuthology.orchestra.run.smithi055.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-12T01:40:19.223 INFO:teuthology.orchestra.run.smithi140.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2023-12-12T01:40:19.242 INFO:teuthology.orchestra.run.smithi055.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-12T01:40:19.242 INFO:teuthology.orchestra.run.smithi055.stdout:uid [ unknown] Ceph.com (release key) 2023-12-12T01:40:19.243 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-12T01:40:19.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-12T01:40:19.251 INFO:teuthology.orchestra.run.smithi140.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2023-12-12T01:40:19.251 INFO:teuthology.orchestra.run.smithi140.stdout:uid [ unknown] Ceph.com (release key) 2023-12-12T01:40:19.251 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2023-12-12T01:40:19.252 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-12T01:40:19.397 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default/ 2023-12-12T01:40:19.397 INFO:teuthology.task.install.deb:Package version is 19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:40:19.484 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default/ 2023-12-12T01:40:19.484 INFO:teuthology.task.install.deb:Package version is 19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:40:19.584 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:40:19.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-12T01:40:19.599 DEBUG:teuthology.orchestra.run.smithi055:> sudo apt-get update 2023-12-12T01:40:19.652 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:40:19.652 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2023-12-12T01:40:19.666 DEBUG:teuthology.orchestra.run.smithi140:> sudo apt-get update 2023-12-12T01:40:19.769 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:40:19.769 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-12T01:40:19.778 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-12T01:40:19.794 INFO:teuthology.orchestra.run.smithi055.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-12T01:40:19.887 INFO:teuthology.orchestra.run.smithi055.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy InRelease 2023-12-12T01:40:19.888 INFO:teuthology.orchestra.run.smithi140.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy InRelease 2023-12-12T01:40:19.910 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-12T01:40:19.923 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2023-12-12T01:40:19.934 INFO:teuthology.orchestra.run.smithi140.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-12T01:40:19.962 INFO:teuthology.orchestra.run.smithi055.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy Release.gpg 2023-12-12T01:40:19.965 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2023-12-12T01:40:19.965 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2023-12-12T01:40:19.991 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2023-12-12T01:40:19.994 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-12T01:40:19.997 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,399 B] 2023-12-12T01:40:20.044 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.7 kB] 2023-12-12T01:40:20.073 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2023-12-12T01:40:20.146 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 28.8 kB in 0s (77.6 kB/s) 2023-12-12T01:40:20.157 INFO:teuthology.orchestra.run.smithi140.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2023-12-12T01:40:20.335 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 28.8 kB in 0s (58.9 kB/s) 2023-12-12T01:40:21.042 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:40:21.058 DEBUG:teuthology.orchestra.run.smithi055:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-130-g3d2db44f-1jammy cephadm=19.0.0-130-g3d2db44f-1jammy ceph-mds=19.0.0-130-g3d2db44f-1jammy ceph-mgr=19.0.0-130-g3d2db44f-1jammy ceph-common=19.0.0-130-g3d2db44f-1jammy ceph-fuse=19.0.0-130-g3d2db44f-1jammy ceph-test=19.0.0-130-g3d2db44f-1jammy ceph-volume=19.0.0-130-g3d2db44f-1jammy radosgw=19.0.0-130-g3d2db44f-1jammy python3-rados=19.0.0-130-g3d2db44f-1jammy python3-rgw=19.0.0-130-g3d2db44f-1jammy python3-cephfs=19.0.0-130-g3d2db44f-1jammy python3-rbd=19.0.0-130-g3d2db44f-1jammy libcephfs2=19.0.0-130-g3d2db44f-1jammy libcephfs-dev=19.0.0-130-g3d2db44f-1jammy librados2=19.0.0-130-g3d2db44f-1jammy librbd1=19.0.0-130-g3d2db44f-1jammy rbd-fuse=19.0.0-130-g3d2db44f-1jammy rbd-mirror=19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:40:21.120 INFO:teuthology.orchestra.run.smithi055.stdout:Reading package lists... 2023-12-12T01:40:21.207 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:40:21.222 DEBUG:teuthology.orchestra.run.smithi140:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-130-g3d2db44f-1jammy cephadm=19.0.0-130-g3d2db44f-1jammy ceph-mds=19.0.0-130-g3d2db44f-1jammy ceph-mgr=19.0.0-130-g3d2db44f-1jammy ceph-common=19.0.0-130-g3d2db44f-1jammy ceph-fuse=19.0.0-130-g3d2db44f-1jammy ceph-test=19.0.0-130-g3d2db44f-1jammy ceph-volume=19.0.0-130-g3d2db44f-1jammy radosgw=19.0.0-130-g3d2db44f-1jammy python3-rados=19.0.0-130-g3d2db44f-1jammy python3-rgw=19.0.0-130-g3d2db44f-1jammy python3-cephfs=19.0.0-130-g3d2db44f-1jammy python3-rbd=19.0.0-130-g3d2db44f-1jammy libcephfs2=19.0.0-130-g3d2db44f-1jammy libcephfs-dev=19.0.0-130-g3d2db44f-1jammy librados2=19.0.0-130-g3d2db44f-1jammy librbd1=19.0.0-130-g3d2db44f-1jammy rbd-fuse=19.0.0-130-g3d2db44f-1jammy rbd-mirror=19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:40:21.283 INFO:teuthology.orchestra.run.smithi140.stdout:Reading package lists... 2023-12-12T01:40:21.304 INFO:teuthology.orchestra.run.smithi055.stdout:Building dependency tree... 2023-12-12T01:40:21.305 INFO:teuthology.orchestra.run.smithi055.stdout:Reading state information... 2023-12-12T01:40:21.472 INFO:teuthology.orchestra.run.smithi140.stdout:Building dependency tree... 2023-12-12T01:40:21.473 INFO:teuthology.orchestra.run.smithi140.stdout:Reading state information... 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-12T01:40:21.485 INFO:teuthology.orchestra.run.smithi055.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-12T01:40:21.486 INFO:teuthology.orchestra.run.smithi055.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:40:21.486 INFO:teuthology.orchestra.run.smithi055.stdout:The following additional packages will be installed: 2023-12-12T01:40:21.486 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-12T01:40:21.486 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-12T01:40:21.487 INFO:teuthology.orchestra.run.smithi055.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-12T01:40:21.487 INFO:teuthology.orchestra.run.smithi055.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-12T01:40:21.488 INFO:teuthology.orchestra.run.smithi055.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: xmlstarlet zip 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout:Suggested packages: 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-12T01:40:21.489 INFO:teuthology.orchestra.run.smithi055.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-12T01:40:21.490 INFO:teuthology.orchestra.run.smithi055.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-12T01:40:21.490 INFO:teuthology.orchestra.run.smithi055.stdout:Recommended packages: 2023-12-12T01:40:21.490 INFO:teuthology.orchestra.run.smithi055.stdout: btrfs-tools 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout:The following NEW packages will be installed: 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-12T01:40:21.597 INFO:teuthology.orchestra.run.smithi055.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-12T01:40:21.598 INFO:teuthology.orchestra.run.smithi055.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-12T01:40:21.599 INFO:teuthology.orchestra.run.smithi055.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-12T01:40:21.601 INFO:teuthology.orchestra.run.smithi055.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-12T01:40:21.601 INFO:teuthology.orchestra.run.smithi055.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-12T01:40:21.601 INFO:teuthology.orchestra.run.smithi055.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-12T01:40:21.602 INFO:teuthology.orchestra.run.smithi055.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-12T01:40:21.603 INFO:teuthology.orchestra.run.smithi055.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi140.stdout:The following packages were automatically installed and are no longer required: 2023-12-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi140.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2023-12-12T01:40:21.645 INFO:teuthology.orchestra.run.smithi140.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2023-12-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi140.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2023-12-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi140.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2023-12-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi140.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2023-12-12T01:40:21.646 INFO:teuthology.orchestra.run.smithi140.stdout:Use 'sudo apt autoremove' to remove them. 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout:The following additional packages will be installed: 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2023-12-12T01:40:21.648 INFO:teuthology.orchestra.run.smithi140.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2023-12-12T01:40:21.649 INFO:teuthology.orchestra.run.smithi140.stdout: xmlstarlet zip 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout:Suggested packages: 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2023-12-12T01:40:21.650 INFO:teuthology.orchestra.run.smithi140.stdout:Recommended packages: 2023-12-12T01:40:21.651 INFO:teuthology.orchestra.run.smithi140.stdout: btrfs-tools 2023-12-12T01:40:21.744 INFO:teuthology.orchestra.run.smithi055.stdout:0 upgraded, 119 newly installed, 0 to remove and 263 not upgraded. 2023-12-12T01:40:21.744 INFO:teuthology.orchestra.run.smithi055.stdout:Need to get 194 MB of archives. 2023-12-12T01:40:21.744 INFO:teuthology.orchestra.run.smithi055.stdout:After this operation, 873 MB of additional disk space will be used. 2023-12-12T01:40:21.744 INFO:teuthology.orchestra.run.smithi055.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-130-g3d2db44f-1jammy [3,510 kB] 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout:The following NEW packages will be installed: 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2023-12-12T01:40:21.754 INFO:teuthology.orchestra.run.smithi140.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2023-12-12T01:40:21.755 INFO:teuthology.orchestra.run.smithi140.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2023-12-12T01:40:21.756 INFO:teuthology.orchestra.run.smithi140.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2023-12-12T01:40:21.757 INFO:teuthology.orchestra.run.smithi140.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2023-12-12T01:40:21.786 INFO:teuthology.orchestra.run.smithi055.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-12T01:40:21.818 INFO:teuthology.orchestra.run.smithi140.stdout:0 upgraded, 119 newly installed, 0 to remove and 263 not upgraded. 2023-12-12T01:40:21.818 INFO:teuthology.orchestra.run.smithi140.stdout:Need to get 194 MB of archives. 2023-12-12T01:40:21.818 INFO:teuthology.orchestra.run.smithi140.stdout:After this operation, 873 MB of additional disk space will be used. 2023-12-12T01:40:21.818 INFO:teuthology.orchestra.run.smithi140.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2023-12-12T01:40:21.919 INFO:teuthology.orchestra.run.smithi140.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-12T01:40:21.922 INFO:teuthology.orchestra.run.smithi140.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-12T01:40:21.943 INFO:teuthology.orchestra.run.smithi140.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-12T01:40:21.977 INFO:teuthology.orchestra.run.smithi140.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-130-g3d2db44f-1jammy [3,510 kB] 2023-12-12T01:40:22.009 INFO:teuthology.orchestra.run.smithi140.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-12T01:40:22.012 INFO:teuthology.orchestra.run.smithi140.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-12T01:40:22.020 INFO:teuthology.orchestra.run.smithi055.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-130-g3d2db44f-1jammy [3,215 kB] 2023-12-12T01:40:22.024 INFO:teuthology.orchestra.run.smithi140.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-12T01:40:22.028 INFO:teuthology.orchestra.run.smithi140.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-12T01:40:22.028 INFO:teuthology.orchestra.run.smithi140.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-12T01:40:22.029 INFO:teuthology.orchestra.run.smithi140.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-12T01:40:22.038 INFO:teuthology.orchestra.run.smithi140.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-12T01:40:22.040 INFO:teuthology.orchestra.run.smithi140.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-12T01:40:22.042 INFO:teuthology.orchestra.run.smithi140.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-12T01:40:22.043 INFO:teuthology.orchestra.run.smithi140.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-12T01:40:22.051 INFO:teuthology.orchestra.run.smithi140.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-12T01:40:22.053 INFO:teuthology.orchestra.run.smithi140.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-12T01:40:22.056 INFO:teuthology.orchestra.run.smithi140.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-12T01:40:22.057 INFO:teuthology.orchestra.run.smithi140.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-12T01:40:22.070 INFO:teuthology.orchestra.run.smithi140.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-12T01:40:22.071 INFO:teuthology.orchestra.run.smithi140.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-12T01:40:22.072 INFO:teuthology.orchestra.run.smithi140.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-12T01:40:22.072 INFO:teuthology.orchestra.run.smithi140.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-12T01:40:22.073 INFO:teuthology.orchestra.run.smithi140.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-12T01:40:22.073 INFO:teuthology.orchestra.run.smithi140.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-12T01:40:22.090 INFO:teuthology.orchestra.run.smithi140.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-12T01:40:22.090 INFO:teuthology.orchestra.run.smithi140.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-12T01:40:22.091 INFO:teuthology.orchestra.run.smithi140.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-12T01:40:22.093 INFO:teuthology.orchestra.run.smithi140.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-12T01:40:22.110 INFO:teuthology.orchestra.run.smithi140.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-12T01:40:22.110 INFO:teuthology.orchestra.run.smithi140.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-12T01:40:22.110 INFO:teuthology.orchestra.run.smithi140.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-12T01:40:22.111 INFO:teuthology.orchestra.run.smithi140.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-12T01:40:22.112 INFO:teuthology.orchestra.run.smithi140.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-12T01:40:22.112 INFO:teuthology.orchestra.run.smithi140.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-12T01:40:22.130 INFO:teuthology.orchestra.run.smithi140.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-12T01:40:22.131 INFO:teuthology.orchestra.run.smithi140.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-12T01:40:22.131 INFO:teuthology.orchestra.run.smithi140.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-12T01:40:22.136 INFO:teuthology.orchestra.run.smithi140.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-12T01:40:22.150 INFO:teuthology.orchestra.run.smithi140.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-12T01:40:22.150 INFO:teuthology.orchestra.run.smithi055.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-130-g3d2db44f-1jammy [710 kB] 2023-12-12T01:40:22.151 INFO:teuthology.orchestra.run.smithi140.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-12T01:40:22.151 INFO:teuthology.orchestra.run.smithi140.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-12T01:40:22.152 INFO:teuthology.orchestra.run.smithi140.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-12T01:40:22.154 INFO:teuthology.orchestra.run.smithi140.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-12T01:40:22.155 INFO:teuthology.orchestra.run.smithi140.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-12T01:40:22.169 INFO:teuthology.orchestra.run.smithi140.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-12T01:40:22.170 INFO:teuthology.orchestra.run.smithi140.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-12T01:40:22.172 INFO:teuthology.orchestra.run.smithi140.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-12T01:40:22.173 INFO:teuthology.orchestra.run.smithi140.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-12T01:40:22.178 INFO:teuthology.orchestra.run.smithi055.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-130-g3d2db44f-1jammy [356 kB] 2023-12-12T01:40:22.189 INFO:teuthology.orchestra.run.smithi140.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-12T01:40:22.191 INFO:teuthology.orchestra.run.smithi140.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-12T01:40:22.197 INFO:teuthology.orchestra.run.smithi055.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-130-g3d2db44f-1jammy [32.8 kB] 2023-12-12T01:40:22.198 INFO:teuthology.orchestra.run.smithi055.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-130-g3d2db44f-1jammy [180 kB] 2023-12-12T01:40:22.206 INFO:teuthology.orchestra.run.smithi055.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-130-g3d2db44f-1jammy [63.5 kB] 2023-12-12T01:40:22.209 INFO:teuthology.orchestra.run.smithi055.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-130-g3d2db44f-1jammy [330 kB] 2023-12-12T01:40:22.213 INFO:teuthology.orchestra.run.smithi055.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2023-12-12T01:40:22.221 INFO:teuthology.orchestra.run.smithi055.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-130-g3d2db44f-1jammy [6,226 kB] 2023-12-12T01:40:22.225 INFO:teuthology.orchestra.run.smithi055.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2023-12-12T01:40:22.249 INFO:teuthology.orchestra.run.smithi140.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-12T01:40:22.250 INFO:teuthology.orchestra.run.smithi140.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-12T01:40:22.252 INFO:teuthology.orchestra.run.smithi140.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-12T01:40:22.252 INFO:teuthology.orchestra.run.smithi140.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-12T01:40:22.253 INFO:teuthology.orchestra.run.smithi140.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-12T01:40:22.278 INFO:teuthology.orchestra.run.smithi140.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-12T01:40:22.280 INFO:teuthology.orchestra.run.smithi140.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-12T01:40:22.280 INFO:teuthology.orchestra.run.smithi140.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-12T01:40:22.305 INFO:teuthology.orchestra.run.smithi140.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-12T01:40:22.305 INFO:teuthology.orchestra.run.smithi140.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-12T01:40:22.306 INFO:teuthology.orchestra.run.smithi140.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-12T01:40:22.306 INFO:teuthology.orchestra.run.smithi140.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-12T01:40:22.307 INFO:teuthology.orchestra.run.smithi140.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-12T01:40:22.307 INFO:teuthology.orchestra.run.smithi055.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2023-12-12T01:40:22.307 INFO:teuthology.orchestra.run.smithi140.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-12T01:40:22.319 INFO:teuthology.orchestra.run.smithi140.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-12T01:40:22.319 INFO:teuthology.orchestra.run.smithi140.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-12T01:40:22.323 INFO:teuthology.orchestra.run.smithi140.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-12T01:40:22.339 INFO:teuthology.orchestra.run.smithi140.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-12T01:40:22.340 INFO:teuthology.orchestra.run.smithi140.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-12T01:40:22.340 INFO:teuthology.orchestra.run.smithi140.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-12T01:40:22.344 INFO:teuthology.orchestra.run.smithi140.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-12T01:40:22.347 INFO:teuthology.orchestra.run.smithi140.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-12T01:40:22.348 INFO:teuthology.orchestra.run.smithi140.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-12T01:40:22.348 INFO:teuthology.orchestra.run.smithi140.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-12T01:40:22.360 INFO:teuthology.orchestra.run.smithi140.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-12T01:40:22.361 INFO:teuthology.orchestra.run.smithi140.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-12T01:40:22.365 INFO:teuthology.orchestra.run.smithi140.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-12T01:40:22.380 INFO:teuthology.orchestra.run.smithi140.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-12T01:40:22.381 INFO:teuthology.orchestra.run.smithi140.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-12T01:40:22.381 INFO:teuthology.orchestra.run.smithi140.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-12T01:40:22.383 INFO:teuthology.orchestra.run.smithi140.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-12T01:40:22.383 INFO:teuthology.orchestra.run.smithi140.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-12T01:40:22.392 INFO:teuthology.orchestra.run.smithi140.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-12T01:40:22.392 INFO:teuthology.orchestra.run.smithi140.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-12T01:40:22.400 INFO:teuthology.orchestra.run.smithi140.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-12T01:40:22.402 INFO:teuthology.orchestra.run.smithi140.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-12T01:40:22.403 INFO:teuthology.orchestra.run.smithi140.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-12T01:40:22.485 INFO:teuthology.orchestra.run.smithi055.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-130-g3d2db44f-1jammy [112 kB] 2023-12-12T01:40:22.487 INFO:teuthology.orchestra.run.smithi055.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-130-g3d2db44f-1jammy [465 kB] 2023-12-12T01:40:22.508 INFO:teuthology.orchestra.run.smithi055.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-130-g3d2db44f-1jammy [22.7 MB] 2023-12-12T01:40:22.537 INFO:teuthology.orchestra.run.smithi055.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2023-12-12T01:40:22.549 INFO:teuthology.orchestra.run.smithi055.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2023-12-12T01:40:22.578 INFO:teuthology.orchestra.run.smithi055.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2023-12-12T01:40:22.588 INFO:teuthology.orchestra.run.smithi055.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2023-12-12T01:40:22.588 INFO:teuthology.orchestra.run.smithi055.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2023-12-12T01:40:22.590 INFO:teuthology.orchestra.run.smithi055.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2023-12-12T01:40:22.602 INFO:teuthology.orchestra.run.smithi140.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-130-g3d2db44f-1jammy [3,215 kB] 2023-12-12T01:40:22.608 INFO:teuthology.orchestra.run.smithi055.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2023-12-12T01:40:22.777 INFO:teuthology.orchestra.run.smithi055.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2023-12-12T01:40:23.023 INFO:teuthology.orchestra.run.smithi140.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-130-g3d2db44f-1jammy [710 kB] 2023-12-12T01:40:23.120 INFO:teuthology.orchestra.run.smithi140.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-130-g3d2db44f-1jammy [356 kB] 2023-12-12T01:40:23.178 INFO:teuthology.orchestra.run.smithi140.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-130-g3d2db44f-1jammy [32.8 kB] 2023-12-12T01:40:23.178 INFO:teuthology.orchestra.run.smithi055.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2023-12-12T01:40:23.186 INFO:teuthology.orchestra.run.smithi140.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-130-g3d2db44f-1jammy [180 kB] 2023-12-12T01:40:23.213 INFO:teuthology.orchestra.run.smithi055.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2023-12-12T01:40:23.215 INFO:teuthology.orchestra.run.smithi140.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-130-g3d2db44f-1jammy [63.5 kB] 2023-12-12T01:40:23.215 INFO:teuthology.orchestra.run.smithi055.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2023-12-12T01:40:23.220 INFO:teuthology.orchestra.run.smithi140.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-130-g3d2db44f-1jammy [330 kB] 2023-12-12T01:40:23.266 INFO:teuthology.orchestra.run.smithi140.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-130-g3d2db44f-1jammy [6,226 kB] 2023-12-12T01:40:23.296 INFO:teuthology.orchestra.run.smithi055.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.1 [174 kB] 2023-12-12T01:40:23.349 INFO:teuthology.orchestra.run.smithi055.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2023-12-12T01:40:23.371 INFO:teuthology.orchestra.run.smithi055.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2023-12-12T01:40:23.382 INFO:teuthology.orchestra.run.smithi055.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2023-12-12T01:40:23.382 INFO:teuthology.orchestra.run.smithi055.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2023-12-12T01:40:23.393 INFO:teuthology.orchestra.run.smithi055.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2023-12-12T01:40:23.394 INFO:teuthology.orchestra.run.smithi055.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2023-12-12T01:40:23.554 INFO:teuthology.orchestra.run.smithi055.stdout:Get:35 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-130-g3d2db44f-1jammy [6,900 kB] 2023-12-12T01:40:23.559 INFO:teuthology.orchestra.run.smithi055.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2023-12-12T01:40:23.643 INFO:teuthology.orchestra.run.smithi055.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2023-12-12T01:40:23.671 INFO:teuthology.orchestra.run.smithi055.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2023-12-12T01:40:23.688 INFO:teuthology.orchestra.run.smithi055.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2023-12-12T01:40:23.732 INFO:teuthology.orchestra.run.smithi055.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2023-12-12T01:40:23.777 INFO:teuthology.orchestra.run.smithi055.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-130-g3d2db44f-1jammy [238 kB] 2023-12-12T01:40:23.785 INFO:teuthology.orchestra.run.smithi055.stdout:Get:42 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-130-g3d2db44f-1jammy [124 kB] 2023-12-12T01:40:23.792 INFO:teuthology.orchestra.run.smithi055.stdout:Get:43 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-130-g3d2db44f-1jammy [1,519 kB] 2023-12-12T01:40:23.855 INFO:teuthology.orchestra.run.smithi055.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-130-g3d2db44f-1jammy [9,958 kB] 2023-12-12T01:40:23.942 INFO:teuthology.orchestra.run.smithi055.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2023-12-12T01:40:23.957 INFO:teuthology.orchestra.run.smithi055.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2023-12-12T01:40:23.964 INFO:teuthology.orchestra.run.smithi055.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2023-12-12T01:40:23.970 INFO:teuthology.orchestra.run.smithi055.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2023-12-12T01:40:23.991 INFO:teuthology.orchestra.run.smithi055.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2023-12-12T01:40:23.995 INFO:teuthology.orchestra.run.smithi055.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2023-12-12T01:40:24.123 INFO:teuthology.orchestra.run.smithi140.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-130-g3d2db44f-1jammy [112 kB] 2023-12-12T01:40:24.133 INFO:teuthology.orchestra.run.smithi140.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-130-g3d2db44f-1jammy [465 kB] 2023-12-12T01:40:24.162 INFO:teuthology.orchestra.run.smithi055.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2023-12-12T01:40:24.201 INFO:teuthology.orchestra.run.smithi140.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-130-g3d2db44f-1jammy [22.7 MB] 2023-12-12T01:40:24.283 INFO:teuthology.orchestra.run.smithi055.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-130-g3d2db44f-1jammy [31.9 MB] 2023-12-12T01:40:24.494 INFO:teuthology.orchestra.run.smithi055.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2023-12-12T01:40:24.526 INFO:teuthology.orchestra.run.smithi055.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2023-12-12T01:40:24.536 INFO:teuthology.orchestra.run.smithi055.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2023-12-12T01:40:24.738 INFO:teuthology.orchestra.run.smithi055.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2023-12-12T01:40:24.739 INFO:teuthology.orchestra.run.smithi055.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2023-12-12T01:40:24.745 INFO:teuthology.orchestra.run.smithi055.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2023-12-12T01:40:24.751 INFO:teuthology.orchestra.run.smithi055.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2023-12-12T01:40:24.766 INFO:teuthology.orchestra.run.smithi055.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2023-12-12T01:40:24.788 INFO:teuthology.orchestra.run.smithi055.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2023-12-12T01:40:24.793 INFO:teuthology.orchestra.run.smithi055.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2023-12-12T01:40:24.969 INFO:teuthology.orchestra.run.smithi055.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2023-12-12T01:40:25.232 INFO:teuthology.orchestra.run.smithi055.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2023-12-12T01:40:25.411 INFO:teuthology.orchestra.run.smithi055.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2023-12-12T01:40:25.425 INFO:teuthology.orchestra.run.smithi055.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2023-12-12T01:40:25.444 INFO:teuthology.orchestra.run.smithi055.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2023-12-12T01:40:25.509 INFO:teuthology.orchestra.run.smithi055.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2023-12-12T01:40:25.848 INFO:teuthology.orchestra.run.smithi055.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2023-12-12T01:40:25.848 INFO:teuthology.orchestra.run.smithi055.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-130-g3d2db44f-1jammy [14.0 kB] 2023-12-12T01:40:25.849 INFO:teuthology.orchestra.run.smithi055.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-130-g3d2db44f-1jammy [865 kB] 2023-12-12T01:40:25.849 INFO:teuthology.orchestra.run.smithi055.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jinja2 all 3.0.3-1 [108 kB] 2023-12-12T01:40:25.852 INFO:teuthology.orchestra.run.smithi055.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2023-12-12T01:40:25.853 INFO:teuthology.orchestra.run.smithi055.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2023-12-12T01:40:25.856 INFO:teuthology.orchestra.run.smithi055.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2023-12-12T01:40:25.866 INFO:teuthology.orchestra.run.smithi055.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-130-g3d2db44f-1jammy [2,180 kB] 2023-12-12T01:40:25.906 INFO:teuthology.orchestra.run.smithi055.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-130-g3d2db44f-1jammy [831 kB] 2023-12-12T01:40:25.917 INFO:teuthology.orchestra.run.smithi055.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-130-g3d2db44f-1jammy [139 kB] 2023-12-12T01:40:25.919 INFO:teuthology.orchestra.run.smithi055.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-130-g3d2db44f-1jammy [1,904 kB] 2023-12-12T01:40:25.954 INFO:teuthology.orchestra.run.smithi055.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-130-g3d2db44f-1jammy [8,625 kB] 2023-12-12T01:40:26.063 INFO:teuthology.orchestra.run.smithi055.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2023-12-12T01:40:26.330 INFO:teuthology.orchestra.run.smithi055.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-130-g3d2db44f-1jammy [14.2 kB] 2023-12-12T01:40:26.331 INFO:teuthology.orchestra.run.smithi055.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-130-g3d2db44f-1jammy [52.2 MB] 2023-12-12T01:40:26.504 INFO:teuthology.orchestra.run.smithi055.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2023-12-12T01:40:26.510 INFO:teuthology.orchestra.run.smithi055.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2023-12-12T01:40:26.801 INFO:teuthology.orchestra.run.smithi055.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2023-12-12T01:40:26.802 INFO:teuthology.orchestra.run.smithi055.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2023-12-12T01:40:26.805 INFO:teuthology.orchestra.run.smithi055.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2023-12-12T01:40:26.807 INFO:teuthology.orchestra.run.smithi055.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2023-12-12T01:40:26.809 INFO:teuthology.orchestra.run.smithi055.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2023-12-12T01:40:26.811 INFO:teuthology.orchestra.run.smithi055.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2023-12-12T01:40:26.812 INFO:teuthology.orchestra.run.smithi055.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2023-12-12T01:40:26.814 INFO:teuthology.orchestra.run.smithi055.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2023-12-12T01:40:26.993 INFO:teuthology.orchestra.run.smithi055.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2023-12-12T01:40:27.250 INFO:teuthology.orchestra.run.smithi140.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-130-g3d2db44f-1jammy [6,900 kB] 2023-12-12T01:40:27.362 INFO:teuthology.orchestra.run.smithi055.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2023-12-12T01:40:27.451 INFO:teuthology.orchestra.run.smithi055.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2023-12-12T01:40:27.476 INFO:teuthology.orchestra.run.smithi055.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2023-12-12T01:40:27.535 INFO:teuthology.orchestra.run.smithi055.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2023-12-12T01:40:27.567 INFO:teuthology.orchestra.run.smithi055.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2023-12-12T01:40:27.591 INFO:teuthology.orchestra.run.smithi055.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2023-12-12T01:40:27.613 INFO:teuthology.orchestra.run.smithi055.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2023-12-12T01:40:27.651 INFO:teuthology.orchestra.run.smithi055.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2023-12-12T01:40:27.672 INFO:teuthology.orchestra.run.smithi055.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2023-12-12T01:40:27.704 INFO:teuthology.orchestra.run.smithi055.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2023-12-12T01:40:27.725 INFO:teuthology.orchestra.run.smithi055.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2023-12-12T01:40:27.748 INFO:teuthology.orchestra.run.smithi055.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2023-12-12T01:40:27.769 INFO:teuthology.orchestra.run.smithi055.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2023-12-12T01:40:27.795 INFO:teuthology.orchestra.run.smithi055.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2023-12-12T01:40:27.817 INFO:teuthology.orchestra.run.smithi055.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2023-12-12T01:40:27.863 INFO:teuthology.orchestra.run.smithi055.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2023-12-12T01:40:27.885 INFO:teuthology.orchestra.run.smithi055.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2023-12-12T01:40:27.905 INFO:teuthology.orchestra.run.smithi140.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-130-g3d2db44f-1jammy [238 kB] 2023-12-12T01:40:27.906 INFO:teuthology.orchestra.run.smithi055.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2023-12-12T01:40:27.932 INFO:teuthology.orchestra.run.smithi140.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-130-g3d2db44f-1jammy [124 kB] 2023-12-12T01:40:27.933 INFO:teuthology.orchestra.run.smithi055.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2023-12-12T01:40:27.946 INFO:teuthology.orchestra.run.smithi140.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-130-g3d2db44f-1jammy [1,519 kB] 2023-12-12T01:40:27.955 INFO:teuthology.orchestra.run.smithi055.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2023-12-12T01:40:28.110 INFO:teuthology.orchestra.run.smithi140.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-130-g3d2db44f-1jammy [9,958 kB] 2023-12-12T01:40:28.971 INFO:teuthology.orchestra.run.smithi055.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-130-g3d2db44f-1jammy [120 kB] 2023-12-12T01:40:28.973 INFO:teuthology.orchestra.run.smithi055.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-130-g3d2db44f-1jammy [38.3 kB] 2023-12-12T01:40:28.975 INFO:teuthology.orchestra.run.smithi055.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-130-g3d2db44f-1jammy [12.3 MB] 2023-12-12T01:40:29.091 INFO:teuthology.orchestra.run.smithi140.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-130-g3d2db44f-1jammy [31.9 MB] 2023-12-12T01:40:29.373 INFO:teuthology.orchestra.run.smithi055.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-130-g3d2db44f-1jammy [91.7 kB] 2023-12-12T01:40:29.375 INFO:teuthology.orchestra.run.smithi055.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-130-g3d2db44f-1jammy [3,124 kB] 2023-12-12T01:40:30.103 INFO:teuthology.orchestra.run.smithi055.stdout:Fetched 194 MB in 8s (24.5 MB/s) 2023-12-12T01:40:30.230 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-12T01:40:30.331 INFO:teuthology.orchestra.run.smithi055.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-12T01:40:30.337 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-12T01:40:30.421 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-12T01:40:30.766 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-12T01:40:30.792 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-12T01:40:30.833 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-12T01:40:30.984 INFO:teuthology.orchestra.run.smithi140.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-130-g3d2db44f-1jammy [14.0 kB] 2023-12-12T01:40:31.058 INFO:teuthology.orchestra.run.smithi140.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-130-g3d2db44f-1jammy [865 kB] 2023-12-12T01:40:31.075 INFO:teuthology.orchestra.run.smithi140.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-130-g3d2db44f-1jammy [2,180 kB] 2023-12-12T01:40:31.114 INFO:teuthology.orchestra.run.smithi140.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-130-g3d2db44f-1jammy [831 kB] 2023-12-12T01:40:31.142 INFO:teuthology.orchestra.run.smithi140.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-130-g3d2db44f-1jammy [139 kB] 2023-12-12T01:40:31.145 INFO:teuthology.orchestra.run.smithi140.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-130-g3d2db44f-1jammy [1,904 kB] 2023-12-12T01:40:31.169 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-12T01:40:31.195 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-12T01:40:31.235 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-12T01:40:31.251 INFO:teuthology.orchestra.run.smithi140.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-130-g3d2db44f-1jammy [8,625 kB] 2023-12-12T01:40:31.614 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-12T01:40:31.641 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:31.680 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:31.836 INFO:teuthology.orchestra.run.smithi140.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-130-g3d2db44f-1jammy [14.2 kB] 2023-12-12T01:40:31.837 INFO:teuthology.orchestra.run.smithi140.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-130-g3d2db44f-1jammy [52.2 MB] 2023-12-12T01:40:32.075 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-12T01:40:32.102 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:32.142 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:32.487 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-12T01:40:32.513 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:32.553 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:32.890 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-12T01:40:32.916 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-12T01:40:32.956 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-12T01:40:33.251 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librados2. 2023-12-12T01:40:33.267 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../007-librados2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:33.301 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:33.745 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librbd1. 2023-12-12T01:40:33.772 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../008-librbd1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:33.812 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:34.240 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libcephfs2. 2023-12-12T01:40:34.266 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:34.306 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:34.575 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rados. 2023-12-12T01:40:34.591 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../010-python3-rados_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:34.625 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:34.737 INFO:teuthology.orchestra.run.smithi140.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-130-g3d2db44f-1jammy [120 kB] 2023-12-12T01:40:34.739 INFO:teuthology.orchestra.run.smithi140.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-130-g3d2db44f-1jammy [38.3 kB] 2023-12-12T01:40:34.740 INFO:teuthology.orchestra.run.smithi140.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-130-g3d2db44f-1jammy [12.3 MB] 2023-12-12T01:40:34.920 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-12T01:40:34.946 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:40:34.986 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:35.205 INFO:teuthology.orchestra.run.smithi140.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-130-g3d2db44f-1jammy [91.7 kB] 2023-12-12T01:40:35.207 INFO:teuthology.orchestra.run.smithi140.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/wip-dis-testing-1/3d2db44fa991bc355074aa9b7647f63b77201fef/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.0.0-130-g3d2db44f-1jammy [3,124 kB] 2023-12-12T01:40:35.239 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cephfs. 2023-12-12T01:40:35.266 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:35.305 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:35.625 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-12T01:40:35.652 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:40:35.700 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:35.903 INFO:teuthology.orchestra.run.smithi140.stdout:Fetched 194 MB in 14s (14.2 MB/s) 2023-12-12T01:40:36.045 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-12T01:40:36.066 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2023-12-12T01:40:36.071 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-12T01:40:36.112 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-12T01:40:36.164 INFO:teuthology.orchestra.run.smithi140.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165989 files and directories currently installed.) 2023-12-12T01:40:36.169 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2023-12-12T01:40:36.265 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-12T01:40:36.414 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-prettytable. 2023-12-12T01:40:36.441 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-12T01:40:36.481 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-12T01:40:36.577 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2023-12-12T01:40:36.603 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2023-12-12T01:40:36.643 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-12T01:40:36.683 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rbd. 2023-12-12T01:40:36.699 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:36.742 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:37.005 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2023-12-12T01:40:37.031 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2023-12-12T01:40:37.078 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-12T01:40:37.079 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-12T01:40:37.105 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-12T01:40:37.136 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-12T01:40:37.389 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-12T01:40:37.404 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-12T01:40:37.447 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-12T01:40:37.550 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libqt5core5a:amd64. 2023-12-12T01:40:37.577 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:37.625 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:37.750 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-12T01:40:37.766 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-12T01:40:37.816 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-12T01:40:38.019 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2023-12-12T01:40:38.046 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:38.086 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:38.153 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua5.1. 2023-12-12T01:40:38.180 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-12T01:40:38.220 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-12T01:40:38.422 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libqt5network5:amd64. 2023-12-12T01:40:38.449 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2023-12-12T01:40:38.481 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:40:38.498 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-any. 2023-12-12T01:40:38.514 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-12T01:40:38.548 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-12T01:40:38.792 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package zip. 2023-12-12T01:40:38.808 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-12T01:40:38.842 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2023-12-12T01:40:38.850 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking zip (3.0-12build2) ... 2023-12-12T01:40:38.869 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2023-12-12T01:40:38.909 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-12T01:40:39.162 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package unzip. 2023-12-12T01:40:39.189 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-12T01:40:39.228 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-12T01:40:39.237 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package librados2. 2023-12-12T01:40:39.263 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../007-librados2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:39.303 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:39.574 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package luarocks. 2023-12-12T01:40:39.601 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-12T01:40:39.640 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-12T01:40:39.773 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package librbd1. 2023-12-12T01:40:39.800 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../008-librbd1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:39.831 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:39.982 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package librgw2. 2023-12-12T01:40:39.998 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../025-librgw2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:40.031 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:40.251 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libcephfs2. 2023-12-12T01:40:40.277 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:40.317 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:40.603 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-rados. 2023-12-12T01:40:40.630 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../010-python3-rados_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:40.669 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:40.776 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rgw. 2023-12-12T01:40:40.792 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:40.834 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:41.162 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-12T01:40:41.189 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-12T01:40:41.214 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-ceph-argparse. 2023-12-12T01:40:41.229 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-12T01:40:41.241 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:40:41.281 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:41.507 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libradosstriper1. 2023-12-12T01:40:41.533 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-cephfs. 2023-12-12T01:40:41.534 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:41.559 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:41.565 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:41.600 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:41.869 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-ceph-common. 2023-12-12T01:40:41.884 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:40:41.919 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:41.968 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-common. 2023-12-12T01:40:41.995 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../029-ceph-common_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:42.035 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:42.189 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-wcwidth. 2023-12-12T01:40:42.215 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2023-12-12T01:40:42.264 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-12T01:40:42.592 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-prettytable. 2023-12-12T01:40:42.618 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2023-12-12T01:40:42.658 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2023-12-12T01:40:42.927 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-rbd. 2023-12-12T01:40:42.954 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:42.994 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:43.297 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package librdkafka1:amd64. 2023-12-12T01:40:43.324 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-base. 2023-12-12T01:40:43.324 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2023-12-12T01:40:43.350 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../030-ceph-base_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:43.355 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-12T01:40:43.400 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:43.633 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libreadline-dev:amd64. 2023-12-12T01:40:43.648 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2023-12-12T01:40:43.691 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2023-12-12T01:40:44.027 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2023-12-12T01:40:44.043 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2023-12-12T01:40:44.069 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-12T01:40:44.085 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-12T01:40:44.086 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-12T01:40:44.119 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-12T01:40:44.380 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package lua5.1. 2023-12-12T01:40:44.396 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2023-12-12T01:40:44.405 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cheroot. 2023-12-12T01:40:44.430 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2023-12-12T01:40:44.432 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-12T01:40:44.489 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-12T01:40:44.742 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package lua-any. 2023-12-12T01:40:44.768 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2023-12-12T01:40:44.783 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-12T01:40:44.808 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking lua-any (27ubuntu1) ... 2023-12-12T01:40:44.810 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-12T01:40:44.850 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-12T01:40:45.060 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package zip. 2023-12-12T01:40:45.076 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2023-12-12T01:40:45.119 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking zip (3.0-12build2) ... 2023-12-12T01:40:45.153 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-12T01:40:45.180 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-12T01:40:45.219 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-12T01:40:45.430 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package unzip. 2023-12-12T01:40:45.457 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.1_amd64.deb ... 2023-12-12T01:40:45.497 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking unzip (6.0-26ubuntu3.1) ... 2023-12-12T01:40:45.497 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-12T01:40:45.524 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-12T01:40:45.564 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-12T01:40:45.926 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package luarocks. 2023-12-12T01:40:45.952 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2023-12-12T01:40:45.992 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2023-12-12T01:40:46.250 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-tempora. 2023-12-12T01:40:46.277 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-12T01:40:46.317 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-12T01:40:46.508 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package librgw2. 2023-12-12T01:40:46.523 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../025-librgw2_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:46.558 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:46.620 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-portend. 2023-12-12T01:40:46.647 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-12T01:40:46.686 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-12T01:40:46.931 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-12T01:40:46.958 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-12T01:40:46.989 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-12T01:40:47.053 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-rgw. 2023-12-12T01:40:47.079 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:47.119 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:47.275 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-12T01:40:47.303 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-12T01:40:47.342 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-12T01:40:47.439 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package liboath0:amd64. 2023-12-12T01:40:47.464 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2023-12-12T01:40:47.505 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2023-12-12T01:40:47.662 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-natsort. 2023-12-12T01:40:47.689 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-12T01:40:47.720 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-12T01:40:47.816 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libradosstriper1. 2023-12-12T01:40:47.832 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:47.874 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:47.999 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-logutils. 2023-12-12T01:40:48.026 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-12T01:40:48.065 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-12T01:40:48.319 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-common. 2023-12-12T01:40:48.345 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../029-ceph-common_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:48.386 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:48.436 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-12T01:40:48.436 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-12T01:40:48.460 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-12T01:40:48.746 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-mako. 2023-12-12T01:40:48.773 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-12T01:40:48.812 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-12T01:40:49.149 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-12T01:40:49.176 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-12T01:40:49.215 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-12T01:40:49.485 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-12T01:40:49.512 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-12T01:40:49.552 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-12T01:40:49.591 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-base. 2023-12-12T01:40:49.617 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../030-ceph-base_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:49.658 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:49.871 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-webob. 2023-12-12T01:40:49.901 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-12T01:40:49.938 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-12T01:40:50.282 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-waitress. 2023-12-12T01:40:50.310 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-12T01:40:50.322 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-jaraco.functools. 2023-12-12T01:40:50.346 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2023-12-12T01:40:50.352 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-12T01:40:50.386 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2023-12-12T01:40:50.669 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-tempita. 2023-12-12T01:40:50.681 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-cheroot. 2023-12-12T01:40:50.696 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-12T01:40:50.707 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2023-12-12T01:40:50.735 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-12T01:40:50.748 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-12T01:40:51.013 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-paste. 2023-12-12T01:40:51.040 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-12T01:40:51.067 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-jaraco.classes. 2023-12-12T01:40:51.080 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-12T01:40:51.094 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2023-12-12T01:40:51.134 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2023-12-12T01:40:51.436 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-jaraco.text. 2023-12-12T01:40:51.458 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-12T01:40:51.462 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2023-12-12T01:40:51.485 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-12T01:40:51.503 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2023-12-12T01:40:51.524 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-12T01:40:51.764 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-jaraco.collections. 2023-12-12T01:40:51.779 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2023-12-12T01:40:51.823 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2023-12-12T01:40:51.827 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-12T01:40:51.854 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-12T01:40:51.894 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-12T01:40:52.125 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-tempora. 2023-12-12T01:40:52.152 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2023-12-12T01:40:52.192 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-tempora (4.1.2-1) ... 2023-12-12T01:40:52.197 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-webtest. 2023-12-12T01:40:52.224 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-12T01:40:52.263 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-12T01:40:52.495 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-portend. 2023-12-12T01:40:52.521 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2023-12-12T01:40:52.561 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-portend (3.0.0-1) ... 2023-12-12T01:40:52.600 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pecan. 2023-12-12T01:40:52.627 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-12T01:40:52.666 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-12T01:40:52.856 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-zc.lockfile. 2023-12-12T01:40:52.882 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2023-12-12T01:40:52.932 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2023-12-12T01:40:53.030 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-certifi. 2023-12-12T01:40:53.057 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-12T01:40:53.096 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-12T01:40:53.226 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-cherrypy3. 2023-12-12T01:40:53.252 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2023-12-12T01:40:53.292 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2023-12-12T01:40:53.416 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-idna. 2023-12-12T01:40:53.443 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-12T01:40:53.499 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-12T01:40:53.646 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-natsort. 2023-12-12T01:40:53.673 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2023-12-12T01:40:53.712 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-natsort (8.0.2-1) ... 2023-12-12T01:40:53.785 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-urllib3. 2023-12-12T01:40:53.801 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-12T01:40:53.835 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-12T01:40:54.024 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-logutils. 2023-12-12T01:40:54.050 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2023-12-12T01:40:54.090 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-logutils (0.3.3-8) ... 2023-12-12T01:40:54.138 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-requests. 2023-12-12T01:40:54.166 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-12T01:40:54.205 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-12T01:40:54.368 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-markupsafe. 2023-12-12T01:40:54.394 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2023-12-12T01:40:54.434 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2023-12-12T01:40:54.474 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-12T01:40:54.490 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-12T01:40:54.524 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-12T01:40:54.704 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-mako. 2023-12-12T01:40:54.730 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2023-12-12T01:40:54.770 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-12T01:40:54.836 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-12T01:40:54.852 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:40:54.894 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:55.733 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-simplegeneric. 2023-12-12T01:40:55.759 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2023-12-12T01:40:55.799 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2023-12-12T01:40:55.966 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-12T01:40:55.993 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:56.032 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:56.102 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-singledispatch. 2023-12-12T01:40:56.129 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2023-12-12T01:40:56.168 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2023-12-12T01:40:56.318 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-12T01:40:56.345 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-12T01:40:56.385 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-12T01:40:56.446 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-webob. 2023-12-12T01:40:56.472 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2023-12-12T01:40:56.512 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2023-12-12T01:40:56.880 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-openssl. 2023-12-12T01:40:56.907 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-12T01:40:56.946 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-12T01:40:57.016 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-waitress. 2023-12-12T01:40:57.042 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2023-12-12T01:40:57.077 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-12T01:40:57.224 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr. 2023-12-12T01:40:57.251 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:57.282 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:57.335 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-tempita. 2023-12-12T01:40:57.361 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2023-12-12T01:40:57.401 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2023-12-12T01:40:57.677 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mon. 2023-12-12T01:40:57.704 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-paste. 2023-12-12T01:40:57.705 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:57.731 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2023-12-12T01:40:57.744 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:57.771 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2023-12-12T01:40:58.124 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2023-12-12T01:40:58.151 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2023-12-12T01:40:58.191 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2023-12-12T01:40:58.493 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pastedeploy. 2023-12-12T01:40:58.520 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2023-12-12T01:40:58.560 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2023-12-12T01:40:58.672 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-osd. 2023-12-12T01:40:58.700 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:40:58.739 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:40:58.863 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-webtest. 2023-12-12T01:40:58.890 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2023-12-12T01:40:58.929 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-webtest (2.0.35-1) ... 2023-12-12T01:40:59.216 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pecan. 2023-12-12T01:40:59.243 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2023-12-12T01:40:59.274 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2023-12-12T01:40:59.587 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-certifi. 2023-12-12T01:40:59.614 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2023-12-12T01:40:59.653 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2023-12-12T01:40:59.990 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-idna. 2023-12-12T01:41:00.017 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2023-12-12T01:41:00.065 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-idna (3.3-1) ... 2023-12-12T01:41:00.367 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-urllib3. 2023-12-12T01:41:00.394 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2023-12-12T01:41:00.434 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-12T01:41:00.609 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph. 2023-12-12T01:41:00.636 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../066-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:00.675 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:00.737 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-requests. 2023-12-12T01:41:00.764 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2023-12-12T01:41:00.804 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-12T01:41:00.995 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-fuse. 2023-12-12T01:41:01.022 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:01.061 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:01.115 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-werkzeug. 2023-12-12T01:41:01.141 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2023-12-12T01:41:01.181 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-12T01:41:01.398 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mds. 2023-12-12T01:41:01.425 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:01.464 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:01.493 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2023-12-12T01:41:01.519 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:01.559 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:01.867 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package cephadm. 2023-12-12T01:41:01.894 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../069-cephadm_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:01.934 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:01.956 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2023-12-12T01:41:01.982 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:02.022 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:02.254 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-12T01:41:02.281 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-12T01:41:02.320 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-12T01:41:02.467 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-bcrypt. 2023-12-12T01:41:02.493 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2023-12-12T01:41:02.533 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2023-12-12T01:41:02.639 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-12T01:41:02.667 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-12T01:41:02.706 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:02.819 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-openssl. 2023-12-12T01:41:02.846 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2023-12-12T01:41:02.886 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-openssl (21.0.0-1) ... 2023-12-12T01:41:03.205 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr. 2023-12-12T01:41:03.232 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:03.272 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:03.684 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mon. 2023-12-12T01:41:03.710 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:03.719 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-babel. 2023-12-12T01:41:03.747 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-12T01:41:03.750 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:03.786 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:04.089 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-jinja2. 2023-12-12T01:41:04.116 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-12T01:41:04.155 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-12T01:41:04.483 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-12T01:41:04.511 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:04.550 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:04.670 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-osd. 2023-12-12T01:41:04.697 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:04.737 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:04.912 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-12T01:41:04.939 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-12T01:41:04.978 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-12T01:41:05.298 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-routes. 2023-12-12T01:41:05.325 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-12T01:41:05.364 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-12T01:41:05.692 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-12T01:41:05.720 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:05.758 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:06.179 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-12T01:41:06.207 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-12T01:41:06.245 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-12T01:41:06.557 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph. 2023-12-12T01:41:06.583 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../066-ceph_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:06.631 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:06.707 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-joblib. 2023-12-12T01:41:06.723 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-12T01:41:06.757 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-12T01:41:06.959 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-fuse. 2023-12-12T01:41:06.986 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:07.025 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:07.094 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-12T01:41:07.121 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-12T01:41:07.160 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-12T01:41:07.395 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mds. 2023-12-12T01:41:07.422 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:07.462 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:07.463 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-sklearn. 2023-12-12T01:41:07.491 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-12T01:41:07.529 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-12T01:41:07.823 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package cephadm. 2023-12-12T01:41:07.849 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../069-cephadm_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:07.890 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:08.178 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-asyncssh. 2023-12-12T01:41:08.180 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-12T01:41:08.202 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2023-12-12T01:41:08.208 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:08.242 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2023-12-12T01:41:08.315 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:08.545 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python-babel-localedata. 2023-12-12T01:41:08.561 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2023-12-12T01:41:08.595 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:09.360 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-cachetools. 2023-12-12T01:41:09.388 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-12T01:41:09.426 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-12T01:41:09.650 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-babel. 2023-12-12T01:41:09.677 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2023-12-12T01:41:09.713 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-12T01:41:09.716 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:09.741 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-12T01:41:09.779 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-12T01:41:10.053 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-jinja2. 2023-12-12T01:41:10.080 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1_all.deb ... 2023-12-12T01:41:10.083 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-12T01:41:10.111 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-12T01:41:10.119 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-jinja2 (3.0.3-1) ... 2023-12-12T01:41:10.141 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-12T01:41:10.419 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-rsa. 2023-12-12T01:41:10.435 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-12T01:41:10.464 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2023-12-12T01:41:10.468 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-12T01:41:10.491 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:10.530 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:10.766 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-google-auth. 2023-12-12T01:41:10.794 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-12T01:41:10.832 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-12T01:41:10.892 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-repoze.lru. 2023-12-12T01:41:10.919 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2023-12-12T01:41:10.958 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2023-12-12T01:41:11.152 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-12T01:41:11.180 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-12T01:41:11.219 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-12T01:41:11.246 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-routes. 2023-12-12T01:41:11.271 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2023-12-12T01:41:11.311 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2023-12-12T01:41:11.506 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-websocket. 2023-12-12T01:41:11.531 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-12T01:41:11.572 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-12T01:41:11.639 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2023-12-12T01:41:11.666 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:11.705 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:11.859 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-12T01:41:11.886 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-12T01:41:11.925 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-12T01:41:12.126 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2023-12-12T01:41:12.153 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2023-12-12T01:41:12.183 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-12T01:41:12.806 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-12T01:41:12.834 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:12.845 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-joblib. 2023-12-12T01:41:12.868 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2023-12-12T01:41:12.897 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:12.912 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2023-12-12T01:41:13.167 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-12T01:41:13.195 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-12T01:41:13.225 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-12T01:41:13.274 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-threadpoolctl. 2023-12-12T01:41:13.301 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2023-12-12T01:41:13.340 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2023-12-12T01:41:13.553 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-12T01:41:13.581 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-12T01:41:13.619 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-12T01:41:13.634 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-sklearn. 2023-12-12T01:41:13.661 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2023-12-12T01:41:13.701 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-12T01:41:13.881 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package jq. 2023-12-12T01:41:13.909 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-12T01:41:13.947 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-12T01:41:14.200 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package socat. 2023-12-12T01:41:14.217 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-12T01:41:14.258 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-12T01:41:14.410 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2023-12-12T01:41:14.437 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:14.536 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package xmlstarlet. 2023-12-12T01:41:14.553 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-12T01:41:14.561 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:14.586 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-12T01:41:14.899 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-test. 2023-12-12T01:41:14.917 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../097-ceph-test_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:14.957 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:15.490 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-cachetools. 2023-12-12T01:41:15.517 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2023-12-12T01:41:15.556 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2023-12-12T01:41:15.853 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pyasn1. 2023-12-12T01:41:15.880 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2023-12-12T01:41:15.919 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2023-12-12T01:41:16.223 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pyasn1-modules. 2023-12-12T01:41:16.250 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2023-12-12T01:41:16.281 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2023-12-12T01:41:16.693 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-rsa. 2023-12-12T01:41:16.719 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2023-12-12T01:41:16.759 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-rsa (4.8-1) ... 2023-12-12T01:41:17.080 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-google-auth. 2023-12-12T01:41:17.106 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2023-12-12T01:41:17.146 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2023-12-12T01:41:17.474 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-requests-oauthlib. 2023-12-12T01:41:17.501 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2023-12-12T01:41:17.520 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package ceph-volume. 2023-12-12T01:41:17.540 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-12T01:41:17.547 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:17.586 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:17.882 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-12T01:41:17.910 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:17.948 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:18.019 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-websocket. 2023-12-12T01:41:18.046 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2023-12-12T01:41:18.085 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-websocket (1.2.3-1) ... 2023-12-12T01:41:18.226 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-12T01:41:18.243 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-12T01:41:18.276 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-12T01:41:18.397 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-kubernetes. 2023-12-12T01:41:18.424 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2023-12-12T01:41:18.463 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-12T01:41:18.573 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-12T01:41:18.600 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-12T01:41:18.641 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-12T01:41:19.001 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package nvme-cli. 2023-12-12T01:41:19.029 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-12T01:41:19.068 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-12T01:41:19.153 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2023-12-12T01:41:19.181 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:19.219 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:19.498 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package pkg-config. 2023-12-12T01:41:19.505 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libonig5:amd64. 2023-12-12T01:41:19.526 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-12T01:41:19.533 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2023-12-12T01:41:19.565 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-12T01:41:19.572 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-12T01:41:19.826 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-12T01:41:19.842 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-12T01:41:19.884 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-12T01:41:19.917 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libjq1:amd64. 2023-12-12T01:41:19.944 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2023-12-12T01:41:19.983 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-12T01:41:20.245 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package jq. 2023-12-12T01:41:20.260 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2023-12-12T01:41:20.296 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-12T01:41:20.303 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2023-12-12T01:41:20.324 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-12T01:41:20.362 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-12T01:41:20.539 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package socat. 2023-12-12T01:41:20.554 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2023-12-12T01:41:20.588 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2023-12-12T01:41:20.632 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pastescript. 2023-12-12T01:41:20.660 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-12T01:41:20.690 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-12T01:41:20.875 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package xmlstarlet. 2023-12-12T01:41:20.892 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2023-12-12T01:41:20.933 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2023-12-12T01:41:20.968 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pluggy. 2023-12-12T01:41:20.996 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-12T01:41:21.068 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-12T01:41:21.280 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-test. 2023-12-12T01:41:21.295 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../097-ceph-test_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:21.330 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:21.380 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-psutil. 2023-12-12T01:41:21.407 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-12T01:41:21.446 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-12T01:41:21.766 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-py. 2023-12-12T01:41:21.794 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-12T01:41:21.832 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-12T01:41:22.160 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pygments. 2023-12-12T01:41:22.189 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-12T01:41:22.227 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-12T01:41:22.706 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-12T01:41:22.734 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-12T01:41:22.781 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-12T01:41:23.075 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-toml. 2023-12-12T01:41:23.103 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-12T01:41:23.141 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-12T01:41:23.427 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-pytest. 2023-12-12T01:41:23.444 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-12T01:41:23.486 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-12T01:41:23.692 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package ceph-volume. 2023-12-12T01:41:23.719 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-130-g3d2db44f-1jammy_all.deb ... 2023-12-12T01:41:23.767 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:23.840 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package python3-simplejson. 2023-12-12T01:41:23.868 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-12T01:41:23.906 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-12T01:41:24.054 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package libcephfs-dev. 2023-12-12T01:41:24.082 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:24.121 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:24.159 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-12T01:41:24.176 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-12T01:41:24.217 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-12T01:41:24.399 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package lua-socket:amd64. 2023-12-12T01:41:24.426 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2023-12-12T01:41:24.465 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-12T01:41:24.795 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package lua-sec:amd64. 2023-12-12T01:41:24.820 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2023-12-12T01:41:24.861 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2023-12-12T01:41:24.921 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package radosgw. 2023-12-12T01:41:24.949 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../116-radosgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:24.987 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:25.249 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package nvme-cli. 2023-12-12T01:41:25.277 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2023-12-12T01:41:25.315 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2023-12-12T01:41:25.732 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-fuse. 2023-12-12T01:41:25.746 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package pkg-config. 2023-12-12T01:41:25.760 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:25.773 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2023-12-12T01:41:25.790 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:25.814 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2023-12-12T01:41:26.048 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python-asyncssh-doc. 2023-12-12T01:41:26.065 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2023-12-12T01:41:26.093 INFO:teuthology.orchestra.run.smithi055.stdout:Selecting previously unselected package rbd-mirror. 2023-12-12T01:41:26.098 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2023-12-12T01:41:26.122 INFO:teuthology.orchestra.run.smithi055.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:26.160 INFO:teuthology.orchestra.run.smithi055.stdout:Unpacking rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:26.494 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-iniconfig. 2023-12-12T01:41:26.509 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2023-12-12T01:41:26.543 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2023-12-12T01:41:26.656 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-12T01:41:26.838 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pastescript. 2023-12-12T01:41:26.865 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2023-12-12T01:41:26.904 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2023-12-12T01:41:26.915 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-12T01:41:27.016 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-12T01:41:27.224 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pluggy. 2023-12-12T01:41:27.252 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2023-12-12T01:41:27.270 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-12T01:41:27.290 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2023-12-12T01:41:27.636 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-psutil. 2023-12-12T01:41:27.640 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-12T01:41:27.663 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2023-12-12T01:41:27.702 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2023-12-12T01:41:28.047 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-py. 2023-12-12T01:41:28.071 INFO:teuthology.orchestra.run.smithi055.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-12T01:41:28.074 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2023-12-12T01:41:28.122 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-py (1.10.0-1) ... 2023-12-12T01:41:28.243 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:28.733 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pygments. 2023-12-12T01:41:28.761 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2023-12-12T01:41:28.800 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2023-12-12T01:41:28.804 INFO:teuthology.orchestra.run.smithi055.stdout:Adding system user cephadm....done 2023-12-12T01:41:28.958 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-12T01:41:29.229 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pyinotify. 2023-12-12T01:41:29.257 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2023-12-12T01:41:29.259 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-12T01:41:29.295 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2023-12-12T01:41:29.518 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-12T01:41:29.573 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-toml. 2023-12-12T01:41:29.601 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2023-12-12T01:41:29.640 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-toml (0.10.2-1) ... 2023-12-12T01:41:29.644 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-12T01:41:29.896 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-12T01:41:29.943 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-pytest. 2023-12-12T01:41:29.970 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2023-12-12T01:41:30.009 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2023-12-12T01:41:30.164 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-12T01:41:30.290 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-12T01:41:30.346 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package python3-simplejson. 2023-12-12T01:41:30.374 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2023-12-12T01:41:30.413 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2023-12-12T01:41:30.608 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-12T01:41:30.690 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package qttranslations5-l10n. 2023-12-12T01:41:30.706 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2023-12-12T01:41:30.748 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2023-12-12T01:41:30.960 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-12T01:41:31.211 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-12T01:41:31.478 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package radosgw. 2023-12-12T01:41:31.479 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-12T01:41:31.506 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../116-radosgw_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:31.544 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:31.747 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-12T01:41:32.015 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-12T01:41:32.266 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-12T01:41:32.297 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package rbd-fuse. 2023-12-12T01:41:32.322 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:32.364 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:32.684 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:32.700 INFO:teuthology.orchestra.run.smithi140.stdout:Selecting previously unselected package rbd-mirror. 2023-12-12T01:41:32.728 INFO:teuthology.orchestra.run.smithi140.stdout:Preparing to unpack .../118-rbd-mirror_19.0.0-130-g3d2db44f-1jammy_amd64.deb ... 2023-12-12T01:41:32.766 INFO:teuthology.orchestra.run.smithi140.stdout:Unpacking rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:32.943 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-12T01:41:33.069 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-12T01:41:33.196 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-12T01:41:33.206 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2023-12-12T01:41:33.346 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-12T01:41:33.464 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2023-12-12T01:41:33.590 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2023-12-12T01:41:33.605 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-12T01:41:33.826 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2023-12-12T01:41:33.997 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-12T01:41:34.029 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-12T01:41:34.084 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-12T01:41:34.194 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2023-12-12T01:41:34.210 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-12T01:41:34.569 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:34.606 INFO:teuthology.orchestra.run.smithi140.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2023-12-12T01:41:34.675 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:34.678 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-12T01:41:34.955 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-12T01:41:35.209 INFO:teuthology.orchestra.run.smithi140.stdout:Adding system user cephadm....done 2023-12-12T01:41:35.231 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-12T01:41:35.332 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2023-12-12T01:41:35.557 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up zip (3.0-12build2) ... 2023-12-12T01:41:35.599 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2023-12-12T01:41:35.683 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-12T01:41:35.851 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2023-12-12T01:41:35.978 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2023-12-12T01:41:36.236 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-repoze.lru (0.7-2) ... 2023-12-12T01:41:36.377 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-12T01:41:36.522 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2023-12-12T01:41:36.636 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-12T01:41:36.639 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-py (1.10.0-1) ... 2023-12-12T01:41:36.754 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-12T01:41:36.880 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-12T01:41:36.932 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2023-12-12T01:41:37.198 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-12T01:41:37.284 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-cachetools (5.0.0-1) ... 2023-12-12T01:41:37.544 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-openssl (21.0.0-1) ... 2023-12-12T01:41:37.583 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-12T01:41:37.825 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-12T01:41:37.837 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up unzip (6.0-26ubuntu3.1) ... 2023-12-12T01:41:38.089 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2023-12-12T01:41:38.219 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-12T01:41:38.331 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2023-12-12T01:41:38.582 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2023-12-12T01:41:38.695 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-12T01:41:38.842 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-ceph-argparse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:39.055 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-12T01:41:39.152 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2023-12-12T01:41:39.277 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2023-12-12T01:41:39.306 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-12T01:41:39.404 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2023-12-12T01:41:39.432 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:39.530 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2023-12-12T01:41:39.742 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-idna (3.3-1) ... 2023-12-12T01:41:39.789 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2023-12-12T01:41:40.077 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-12T01:41:40.184 INFO:teuthology.orchestra.run.smithi140.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2023-12-12T01:41:40.221 INFO:teuthology.orchestra.run.smithi140.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2023-12-12T01:41:40.284 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2023-12-12T01:41:40.419 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2023-12-12T01:41:40.820 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:40.946 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-natsort (8.0.2-1) ... 2023-12-12T01:41:41.222 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2023-12-12T01:41:41.315 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-12T01:41:41.481 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2023-12-12T01:41:41.567 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:41.701 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-12T01:41:41.774 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up zip (3.0-12build2) ... 2023-12-12T01:41:41.900 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2023-12-12T01:41:41.961 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-12T01:41:42.095 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-12T01:41:42.238 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-12T01:41:42.556 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-12T01:41:42.586 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2023-12-12T01:41:42.816 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-12T01:41:42.846 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2023-12-12T01:41:42.972 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2023-12-12T01:41:43.058 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:43.098 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2023-12-12T01:41:43.185 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-12T01:41:43.399 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2023-12-12T01:41:43.486 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-12T01:41:43.745 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-12T01:41:43.767 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-certifi (2020.6.20-1) ... 2023-12-12T01:41:44.013 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-12T01:41:44.273 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-12T01:41:44.326 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2023-12-12T01:41:44.599 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-12T01:41:44.703 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2023-12-12T01:41:44.858 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-12T01:41:44.976 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-12T01:41:45.021 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2023-12-12T01:41:45.252 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-12T01:41:45.347 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2023-12-12T01:41:45.353 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-12T01:41:45.590 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2023-12-12T01:41:45.663 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-12T01:41:45.716 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:45.964 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-12T01:41:46.009 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-idna (3.3-1) ... 2023-12-12T01:41:46.232 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-12T01:41:46.328 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2023-12-12T01:41:46.341 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-12T01:41:46.451 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:46.538 INFO:teuthology.orchestra.run.smithi055.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-12T01:41:46.785 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-12T01:41:46.911 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-12T01:41:47.305 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-12T01:41:47.566 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2023-12-12T01:41:47.573 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-12T01:41:47.698 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-12T01:41:47.818 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:47.950 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:47.952 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-toml (0.10.2-1) ... 2023-12-12T01:41:48.076 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-12T01:41:48.212 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2023-12-12T01:41:48.338 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2023-12-12T01:41:48.435 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-12T01:41:48.464 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2023-12-12T01:41:48.761 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-12T01:41:48.765 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2023-12-12T01:41:49.025 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2023-12-12T01:41:49.037 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-12T01:41:49.164 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-12T01:41:49.327 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:49.440 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-12T01:41:49.478 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2023-12-12T01:41:49.733 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-12T01:41:49.821 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2023-12-12T01:41:50.072 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-logutils (0.3.3-8) ... 2023-12-12T01:41:50.078 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-12T01:41:50.323 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-tempora (4.1.2-1) ... 2023-12-12T01:41:50.471 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-12T01:41:50.582 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2023-12-12T01:41:50.772 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-12T01:41:50.834 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-prettytable (2.5.0-2) ... 2023-12-12T01:41:51.108 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-12T01:41:51.110 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2023-12-12T01:41:51.234 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:51.261 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-websocket (1.2.3-1) ... 2023-12-12T01:41:51.352 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:51.461 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-12T01:41:51.562 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2023-12-12T01:41:51.671 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2023-12-12T01:41:51.781 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:51.915 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:51.948 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2023-12-12T01:41:52.041 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-12T01:41:52.249 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2023-12-12T01:41:52.318 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:52.444 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:52.517 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2023-12-12T01:41:52.570 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:52.651 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2023-12-12T01:41:52.777 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2023-12-12T01:41:52.788 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-12T01:41:52.788 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-12T01:41:52.866 INFO:teuthology.orchestra.run.smithi140.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2023-12-12T01:41:53.095 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2023-12-12T01:41:53.222 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2023-12-12T01:41:53.305 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:53.423 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:53.606 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2023-12-12T01:41:53.858 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up lua-any (27ubuntu1) ... 2023-12-12T01:41:53.908 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:53.984 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-portend (3.0.0-1) ... 2023-12-12T01:41:54.059 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-12T01:41:54.236 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2023-12-12T01:41:54.361 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-jinja2 (3.0.3-1) ... 2023-12-12T01:41:54.670 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2023-12-12T01:41:54.963 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2023-12-12T01:41:55.231 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2023-12-12T01:41:55.346 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:55.365 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-rsa (4.8-1) ... 2023-12-12T01:41:55.464 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:55.565 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:55.625 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-webtest (2.0.35-1) ... 2023-12-12T01:41:55.666 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:55.783 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:55.893 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2023-12-12T01:41:56.255 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2023-12-12T01:41:56.326 INFO:teuthology.orchestra.run.smithi055.stdout:Adding group ceph....done 2023-12-12T01:41:56.623 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pastescript (2.0.2-4) ... 2023-12-12T01:41:56.670 INFO:teuthology.orchestra.run.smithi055.stdout:Adding system user ceph....done 2023-12-12T01:41:56.838 INFO:teuthology.orchestra.run.smithi055.stdout:Setting system user ceph properties....done 2023-12-12T01:41:56.850 INFO:teuthology.orchestra.run.smithi055.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-12T01:41:56.908 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2023-12-12T01:41:56.996 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-12T01:41:57.244 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2023-12-12T01:41:57.337 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-12T01:41:57.387 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up librados2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:57.521 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libsqlite3-mod-ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:57.648 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2023-12-12T01:41:57.785 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:57.912 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.009 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libcephfs2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.135 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libradosstriper1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.303 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-google-auth (1.5.1-3) ... 2023-12-12T01:41:58.375 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-12T01:41:58.375 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-12T01:41:58.596 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up librbd1 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.722 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr-modules-core (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.829 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:58.840 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:59.036 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-12T01:41:59.036 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2023-12-12T01:41:59.491 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up libcephfs-dev (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:59.548 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-12T01:41:59.611 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-rados (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:59.737 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up librgw2 (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:41:59.863 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2023-12-12T01:42:00.008 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:00.230 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-12T01:42:00.230 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-12T01:42:00.693 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-12T01:42:00.909 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-12T01:42:01.000 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-rbd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.126 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up rbd-fuse (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.252 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-rgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.386 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.386 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up python3-cephfs (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.513 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-common (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:01.604 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-12T01:42:01.604 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-12T01:42:02.022 INFO:teuthology.orchestra.run.smithi140.stdout:Adding group ceph....done 2023-12-12T01:42:02.079 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:02.362 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-12T01:42:02.362 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-12T01:42:02.424 INFO:teuthology.orchestra.run.smithi140.stdout:Adding system user ceph....done 2023-12-12T01:42:02.600 INFO:teuthology.orchestra.run.smithi140.stdout:Setting system user ceph properties....done 2023-12-12T01:42:02.614 INFO:teuthology.orchestra.run.smithi140.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2023-12-12T01:42:02.749 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2023-12-12T01:42:02.847 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:02.957 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:03.063 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2023-12-12T01:42:03.083 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:03.301 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-12T01:42:03.302 INFO:teuthology.orchestra.run.smithi055.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-12T01:42:03.530 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-test (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:03.656 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up radosgw (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:03.784 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:03.910 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:04.019 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:04.170 INFO:teuthology.orchestra.run.smithi055.stdout:Setting up ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:04.207 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-12T01:42:04.207 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2023-12-12T01:42:04.604 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-12T01:42:04.721 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-12T01:42:04.874 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-12T01:42:04.884 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-base (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:05.235 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2023-12-12T01:42:05.678 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mds (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:05.909 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-12T01:42:05.909 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2023-12-12T01:42:05.968 INFO:teuthology.orchestra.run.smithi055.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-12T01:42:06.354 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up rbd-mirror (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:06.570 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-12T01:42:06.570 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2023-12-12T01:42:06.673 INFO:teuthology.orchestra.run.smithi055.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-12T01:42:06.679 DEBUG:teuthology.parallel:result is None 2023-12-12T01:42:07.014 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:07.225 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-12T01:42:07.225 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2023-12-12T01:42:07.698 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-osd (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:07.973 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-12T01:42:07.973 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2023-12-12T01:42:08.483 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr-k8sevents (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:08.609 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:08.760 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mon (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:08.971 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-12T01:42:08.971 INFO:teuthology.orchestra.run.smithi140.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2023-12-12T01:42:09.436 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr-cephadm (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:09.579 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:09.852 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-mgr-dashboard (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:10.001 INFO:teuthology.orchestra.run.smithi140.stdout:Setting up ceph-volume (19.0.0-130-g3d2db44f-1jammy) ... 2023-12-12T01:42:10.346 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2023-12-12T01:42:10.455 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2023-12-12T01:42:10.599 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for man-db (2.10.2-1) ... 2023-12-12T01:42:11.708 INFO:teuthology.orchestra.run.smithi140.stdout:Processing triggers for install-info (6.8-4build1) ... 2023-12-12T01:42:12.370 INFO:teuthology.orchestra.run.smithi140.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2023-12-12T01:42:12.374 DEBUG:teuthology.parallel:result is None 2023-12-12T01:42:12.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-12T01:42:12.525 DEBUG:teuthology.orchestra.run.smithi055:> dpkg-query -W -f '${Version}' ceph 2023-12-12T01:42:12.546 INFO:teuthology.orchestra.run.smithi055.stdout:19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:42:12.547 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:42:12.547 INFO:teuthology.task.install:The correct ceph version 19.0.0-130-g3d2db44f-1jammy is installed. 2023-12-12T01:42:12.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=3d2db44fa991bc355074aa9b7647f63b77201fef 2023-12-12T01:42:12.763 DEBUG:teuthology.orchestra.run.smithi140:> dpkg-query -W -f '${Version}' ceph 2023-12-12T01:42:12.782 INFO:teuthology.orchestra.run.smithi140.stdout:19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:42:12.783 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-130-g3d2db44f-1jammy 2023-12-12T01:42:12.783 INFO:teuthology.task.install:The correct ceph version 19.0.0-130-g3d2db44f-1jammy is installed. 2023-12-12T01:42:12.785 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-12T01:42:12.786 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:12.786 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-12T01:42:12.801 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:12.801 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-12T01:42:12.836 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-12T01:42:12.836 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:12.837 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/daemon-helper 2023-12-12T01:42:12.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-12T01:42:12.911 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:12.911 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/daemon-helper 2023-12-12T01:42:12.924 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-12T01:42:12.980 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-12T01:42:12.980 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:12.980 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-12T01:42:12.992 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-12T01:42:13.047 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:13.047 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-12T01:42:13.060 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-12T01:42:13.115 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-12T01:42:13.115 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:13.115 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/usr/bin/stdin-killer 2023-12-12T01:42:13.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-12T01:42:13.183 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:13.183 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/usr/bin/stdin-killer 2023-12-12T01:42:13.196 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-12T01:42:13.250 INFO:teuthology.run_tasks:Running task ceph... 2023-12-12T01:42:13.318 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-12-12T01:42:13.318 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 777 /var/log/ceph 2023-12-12T01:42:13.320 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 777 /var/log/ceph 2023-12-12T01:42:13.333 INFO:tasks.ceph:Disabling ceph logrotate... 2023-12-12T01:42:13.333 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-12T01:42:13.378 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/logrotate.d/ceph 2023-12-12T01:42:13.390 INFO:tasks.ceph:Creating extra log directories... 2023-12-12T01:42:13.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-12T01:42:13.432 DEBUG:teuthology.orchestra.run.smithi140:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-12-12T01:42:13.458 INFO:tasks.ceph:Creating ceph cluster cluster1... 2023-12-12T01:42:13.458 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-12T01:42:13.458 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'branch': 'wip-dis-testing-1', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7488100', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-1'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-2', 'sha1': 'e7f1b79d320b95939278401b2a5bd1b491f81fe3'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7033, 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'sleep_before_teardown': 0, 'subset': '91/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'e7f1b79d320b95939278401b2a5bd1b491f81fe3', 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhDAJKfrLVA+K4Ad6CRbRTxf9G11RQdKVLaVWOow2atgabyCvvM6RPcIaCoqkzXKn0+HB/ochBaexdCmpXV9EQ=', 'smithi140.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRz2LD1q4ZzlkncmnwebNaW2K130qCITj6zPV4RfGgzvT1s4SG2i00T3UnQlVQKmsySsbKYfD5Kvf2GQXBRRzs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-12_01:13:29', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-12T01:42:13.459 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-12T01:42:13.488 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2023-12-12T01:42:13.494 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2023-12-12T01:42:13.536 DEBUG:teuthology.orchestra.run.smithi140:> sudo install -d -m0777 -- /var/run/ceph 2023-12-12T01:42:13.549 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:13.549 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/scratch_devs of=/dev/stdout 2023-12-12T01:42:13.595 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-12T01:42:13.596 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_1 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 863 Links: 1 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-12 01:36:49.394267746 +0000 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-12 01:36:48.994307739 +0000 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-12 01:36:48.994307739 +0000 2023-12-12T01:42:13.644 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2023-12-12T01:42:13.645 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-12T01:42:13.698 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-12T01:42:13.698 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-12T01:42:13.698 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000502376 s, 1.0 MB/s 2023-12-12T01:42:13.699 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-12T01:42:13.749 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_2 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 875 Links: 1 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-12 01:36:50.010206218 +0000 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-12 01:36:49.638243366 +0000 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-12 01:36:49.638243366 +0000 2023-12-12T01:42:13.800 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2023-12-12T01:42:13.800 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-12T01:42:13.854 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-12T01:42:13.854 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-12T01:42:13.854 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000405924 s, 1.3 MB/s 2023-12-12T01:42:13.855 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-12T01:42:13.905 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_3 2023-12-12T01:42:13.951 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 890 Links: 1 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-12 01:36:50.598149229 +0000 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-12 01:36:50.254182569 +0000 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-12 01:36:50.254182569 +0000 2023-12-12T01:42:13.952 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2023-12-12T01:42:13.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-12T01:42:14.006 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-12T01:42:14.006 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-12T01:42:14.006 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000480994 s, 1.1 MB/s 2023-12-12T01:42:14.007 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-12T01:42:14.056 DEBUG:teuthology.orchestra.run.smithi055:> stat /dev/vg_nvme/lv_4 2023-12-12T01:42:14.104 INFO:teuthology.orchestra.run.smithi055.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-12T01:42:14.104 INFO:teuthology.orchestra.run.smithi055.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:Device: 5h/5d Inode: 908 Links: 1 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:Access: 2023-12-12 01:36:51.218089569 +0000 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:Modify: 2023-12-12 01:36:50.862123641 +0000 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout:Change: 2023-12-12 01:36:50.862123641 +0000 2023-12-12T01:42:14.105 INFO:teuthology.orchestra.run.smithi055.stdout: Birth: - 2023-12-12T01:42:14.106 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-12T01:42:14.158 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records in 2023-12-12T01:42:14.159 INFO:teuthology.orchestra.run.smithi055.stderr:1+0 records out 2023-12-12T01:42:14.159 INFO:teuthology.orchestra.run.smithi055.stderr:512 bytes copied, 0.000440338 s, 1.2 MB/s 2023-12-12T01:42:14.160 DEBUG:teuthology.orchestra.run.smithi055:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-12T01:42:14.209 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:14.209 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-12T01:42:14.209 INFO:tasks.ceph:Generating config... 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client] log to stderr = False 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-12T01:42:14.210 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.211 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.211 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-12T01:42:14.211 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-12T01:42:14.211 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-12T01:42:14.211 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-12T01:42:14.211 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-12T01:42:14.211 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-12T01:42:14.211 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-12T01:42:14.211 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-12T01:42:14.211 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-12T01:42:14.212 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-12T01:42:14.212 INFO:tasks.ceph:Setting up cluster1.mon.a... 2023-12-12T01:42:14.213 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.282 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.308 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.391 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.447 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.55')] 2023-12-12T01:42:14.448 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.55', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-12T01:42:14.449 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:14.449 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-12T01:42:14.496 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.55 --print /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:14.575 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: generated fsid f2278db0-8e6c-4cd2-a0a7-022133e15780 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:setting min_mon_release = pacific 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:epoch 0 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:fsid f2278db0-8e6c-4cd2-a0a7-022133e15780 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:last_changed 2023-12-12T01:42:14.578921+0000 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:created 2023-12-12T01:42:14.578921+0000 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:min_mon_release 16 (pacific) 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:election_strategy: 1 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:0: [v2:172.21.15.55:3300/0,v1:172.21.15.55:6789/0] mon.a 2023-12-12T01:42:14.576 INFO:teuthology.orchestra.run.smithi055.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2023-12-12T01:42:14.577 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-12T01:42:14.624 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID f2278db0-8e6c-4cd2-a0a7-022133e15780... 2023-12-12T01:42:14.625 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-12T01:42:14.668 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2023-12-12T01:42:14.697 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2023-12-12T01:42:14.697 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2023-12-12T01:42:14.697 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-12T01:42:14.697 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2023-12-12T01:42:14.697 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.698 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2023-12-12T01:42:14.699 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = 172.21.15.55 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon client directed command retry = 5 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on pool no app = False 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: ms inject socket failures = 5000 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = f2278db0-8e6c-4cd2-a0a7-022133e15780 2023-12-12T01:42:14.700 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2023-12-12T01:42:14.701 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2023-12-12T01:42:14.702 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: bdev async discard = True 2023-12-12T01:42:14.703 INFO:teuthology.orchestra.run.smithi055.stdout: bdev enable discard = True 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore allocator = stupid 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 1/20 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 1/20 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 4/10 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2023-12-12T01:42:14.704 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-12-12T01:42:14.705 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.706 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2023-12-12T01:42:14.707 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: debug rbd = 20 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: debug rbd_mirror = 15 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: log to stderr = False 2023-12-12T01:42:14.708 INFO:teuthology.orchestra.run.smithi055.stdout: rbd default features = 125 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: rbd mirror image policy type = simple 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.0] 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.1] 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.2] 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.709 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.710 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.3] 2023-12-12T01:42:14.710 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.710 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout:[global] 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: chdir = "" 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: auth supported = cephx 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: filestore xattr use omap = true 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: mon clock drift allowed = 1.000 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: osd crush chooseleaf type = 0 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: auth debug = true 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.712 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on old message = true 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on bug = true 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon pg warn max object skew = 0 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: # disable pg_autoscaler by default for new pools 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default size = 2 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow primary affinity = true 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow pg remap = true 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on legacy crush tunables = false 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on crush straw calc version zero = false 2023-12-12T01:42:14.713 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on no sortbitwise = false 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on osd down out interval zero = false 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on too few osds = false 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon_allow_pool_size_one = true 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: osd default data pool replay window = 5 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon allow pool delete = true 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon cluster log file level = debug 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: debug asserts on shutdown = true 2023-12-12T01:42:14.714 INFO:teuthology.orchestra.run.smithi140.stdout: mon health detail to clog = false 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: mon host = 172.21.15.55 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: mon client directed command retry = 5 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on pool no app = False 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: ms inject socket failures = 5000 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: fsid = f2278db0-8e6c-4cd2-a0a7-022133e15780 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout:[osd] 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd journal size = 100 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub load threshold = 5.0 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub max interval = 600 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd mclock profile = high_recovery_ops 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd recover clone overlap = true 2023-12-12T01:42:14.715 INFO:teuthology.orchestra.run.smithi140.stdout: osd recovery max chunk = 1048576 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug shutdown = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug op order = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug verify stray on activate = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug trim objects = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd open classes on start = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug pg log writeout = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd deep scrub update digest min age = 30 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: osd map max advance = 10 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: journal zero on create = true 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.716 INFO:teuthology.orchestra.run.smithi140.stdout: filestore ondisk finisher threads = 3 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: filestore apply finisher threads = 3 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bdev debug aio = true 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug misdirected ops = true 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bdev async discard = True 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bdev enable discard = True 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore allocator = stupid 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore block size = 96636764160 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore fsck on mount = True 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluefs = 1/20 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluestore = 1/20 2023-12-12T01:42:14.717 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:14.718 INFO:teuthology.orchestra.run.smithi140.stdout: debug osd = 20 2023-12-12T01:42:14.718 INFO:teuthology.orchestra.run.smithi140.stdout: debug rocksdb = 4/10 2023-12-12T01:42:14.718 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:14.718 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd full ratio = 0.9 2023-12-12T01:42:14.718 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd nearfull ratio = 0.8 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: osd failsafe full ratio = 0.95 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: osd objectstore = bluestore 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout:[mgr] 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: debug mgr = 20 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: mgr/telemetry/nag = false 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout:[mon] 2023-12-12T01:42:14.719 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: debug paxos = 20 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon data avail warn = 5 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon mgr mkfs grace = 240 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd reporter subtree level = osd 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd prime pg temp = true 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: auth mon ticket ttl = 660 # 11m 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: auth service ticket ttl = 240 # 4m 2023-12-12T01:42:14.720 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: # don't complain about insecure global_id in the test suite 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: # 1m isn't quite enough 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: mon_down_mkfs_grace = 2m 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_filestore_osds = false 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout:[client] 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: rgw cache enabled = true 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable ops log = true 2023-12-12T01:42:14.721 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable usage log = true 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: debug rbd = 20 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: debug rbd_mirror = 15 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: log to stderr = False 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: rbd default features = 125 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: rbd mirror image policy type = simple 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout:[mon.a] 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.0] 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.1] 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.722 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.2] 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.3] 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:14.723 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:14.723 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2023-12-12T01:42:14.723 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.804 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-12T01:42:14.805 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:14.805 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2023-12-12T01:42:14.810 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:14.810 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2023-12-12T01:42:14.855 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:42:14.855 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:14.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-12T01:42:14.855 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.921 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:14.921 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:14.971 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:42:14.972 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:14.972 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/cluster1.keyring 2023-12-12T01:42:14.972 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2023-12-12T01:42:14.996 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:14.996 DEBUG:teuthology.orchestra.run.smithi140:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:15.044 INFO:tasks.ceph:Setting up mon nodes... 2023-12-12T01:42:15.045 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-12T01:42:15.045 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-12T01:42:15.089 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2023-12-12T01:42:15.091 INFO:tasks.ceph:Setting up mds nodes... 2023-12-12T01:42:15.092 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-12T01:42:15.092 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2023-12-12T01:42:15.173 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/cluster1.client.0.keyring 2023-12-12T01:42:15.182 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2023-12-12T01:42:15.218 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2023-12-12T01:42:15.267 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-12T01:42:15.345 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2023-12-12T01:42:15.355 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-12T01:42:15.431 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2023-12-12T01:42:15.440 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-12T01:42:15.514 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2023-12-12T01:42:15.524 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-12T01:42:15.597 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2023-12-12T01:42:15.609 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-12T01:42:15.681 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2023-12-12T01:42:15.691 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-12T01:42:15.764 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2023-12-12T01:42:15.774 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-12T01:42:15.844 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2023-12-12T01:42:15.853 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-12T01:42:15.853 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-12T01:42:15.854 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2023-12-12T01:42:15.868 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:15.868 INFO:tasks.ceph:role: cluster1.osd.0 2023-12-12T01:42:15.868 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:42:15.868 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:15.954 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:15.955 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:15.955 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:17.252 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2023-12-12T01:42:17.255 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2023-12-12T01:42:17.255 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2023-12-12T01:42:17.317 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2023-12-12T01:42:17.368 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:17.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:17.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2023-12-12T01:42:17.422 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:17.422 INFO:tasks.ceph:role: cluster1.osd.1 2023-12-12T01:42:17.422 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:42:17.422 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:17.503 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:17.504 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:17.504 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:17.504 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:17.504 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:18.802 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2023-12-12T01:42:18.804 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2023-12-12T01:42:18.804 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2023-12-12T01:42:18.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2023-12-12T01:42:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:18.925 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:18.925 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2023-12-12T01:42:18.979 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:18.979 INFO:tasks.ceph:role: cluster1.osd.2 2023-12-12T01:42:18.979 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:42:18.979 DEBUG:teuthology.orchestra.run.smithi055:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-12T01:42:19.066 INFO:teuthology.orchestra.run.smithi055.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:19.066 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:19.067 INFO:teuthology.orchestra.run.smithi055.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:20.367 INFO:teuthology.orchestra.run.smithi055.stdout:Discarding blocks...Done. 2023-12-12T01:42:20.369 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi055.front.sepia.ceph.com -o noatime 2023-12-12T01:42:20.369 DEBUG:teuthology.orchestra.run.smithi055:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2023-12-12T01:42:20.431 DEBUG:teuthology.orchestra.run.smithi055:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2023-12-12T01:42:20.486 INFO:teuthology.orchestra.run.smithi055.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:20.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:20.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:20.604 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2023-12-12T01:42:20.605 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2023-12-12T01:42:20.605 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-12T01:42:20.605 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-12T01:42:20.605 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 bluestore(/var/lib/ceph/osd/cluster1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-0/block: (2) No such file or directory 2023-12-12T01:42:20.606 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:20.605+0000 7f01f8c3c640 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2023-12-12T01:42:23.632 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2023-12-12T01:42:23.646 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:23.731 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2023-12-12T01:42:23.732 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2023-12-12T01:42:23.732 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-12T01:42:23.732 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-12T01:42:23.732 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 bluestore(/var/lib/ceph/osd/cluster1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-1/block: (2) No such file or directory 2023-12-12T01:42:23.733 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:23.733+0000 7fa6f0352640 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2023-12-12T01:42:26.753 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2023-12-12T01:42:26.808 DEBUG:teuthology.orchestra.run.smithi055:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:26.891 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2023-12-12T01:42:26.892 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2023-12-12T01:42:26.892 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-12T01:42:26.893 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-12T01:42:26.893 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster1-2/block: (2) No such file or directory 2023-12-12T01:42:26.893 INFO:teuthology.orchestra.run.smithi055.stderr:2023-12-12T01:42:26.893+0000 7fb37161a640 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2023-12-12T01:42:29.914 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2023-12-12T01:42:29.968 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-12T01:42:29.968 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:29.969 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2023-12-12T01:42:30.023 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:30.023 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2023-12-12T01:42:30.078 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:30.079 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2023-12-12T01:42:30.135 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:30.135 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2023-12-12T01:42:30.192 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:30.192 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2023-12-12T01:42:30.239 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.239 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2023-12-12T01:42:30.246 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.246 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2023-12-12T01:42:30.292 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.292 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2023-12-12T01:42:30.340 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.340 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2023-12-12T01:42:30.388 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.388 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2023-12-12T01:42:30.436 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.436 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2023-12-12T01:42:30.484 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.484 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2023-12-12T01:42:30.532 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:30.532 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2023-12-12T01:42:30.580 INFO:tasks.ceph:Adding keys to all mons... 2023-12-12T01:42:30.580 DEBUG:teuthology.orchestra.run.smithi055:> sudo tee -a /etc/ceph/cluster1.keyring 2023-12-12T01:42:30.592 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr.x] 2023-12-12T01:42:30.592 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undl8G2BBRAAxDo7jiHlbsiCsv17BRJLbw== 2023-12-12T01:42:30.592 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.0] 2023-12-12T01:42:30.592 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB8undljm88JBAAReq76+Hlf24JKmBSaClYiQ== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.1] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB/undlXu7IKxAAtefyHkhh+noUUQgt3ACaBA== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[osd.2] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQCCundlAmBYNRAADqw7Y5ay5lhR53q9X/H7mg== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.0] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlbxt/ChAAL/fp5MPiTWLpNoOl2zCcJQ== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlPy07DRAAmYOdFkgOJOHsj2QkR48ftg== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.0] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlyxfDFBAAIMgdzCS6rjxixlP3rEKKtw== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.1] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undl+JPcGRAAv3Cb03jeqTuWi6ig6eTlyg== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.2] 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlITzfHhAABUbJTfVxjeIpGT2dP31sEA== 2023-12-12T01:42:30.593 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.3] 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlrH/VIxAAKQUidHishFwb8kV8FSsW0g== 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.4] 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlia7OKBAAJQFWR2PLiArVE9zgoxsYGQ== 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.5] 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undlEF7ALRAA4z1hEEwem4qLNlIarCIk7Q== 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.6] 2023-12-12T01:42:30.594 INFO:teuthology.orchestra.run.smithi055.stdout: key = AQB3undl8s6GMhAAYoPIgnUzrhAVAH1UIW/FqQ== 2023-12-12T01:42:30.598 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-12T01:42:30.686 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:42:30.731 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:42:30.821 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:42:30.909 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:30.998 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.089 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.174 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.263 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.351 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.439 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.527 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:42:31.574 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-12T01:42:31.574 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2023-12-12T01:42:31.584 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2023-12-12T01:42:31.681 DEBUG:teuthology.orchestra.run.smithi055:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2023-12-12T01:42:31.692 DEBUG:teuthology.orchestra.run.smithi055:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2023-12-12T01:42:31.738 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2023-12-12T01:42:31.739 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2023-12-12T01:42:31.739 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2023-12-12T01:42:31.784 INFO:tasks.ceph.cluster1.mon.a:Started 2023-12-12T01:42:31.785 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2023-12-12T01:42:31.785 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2023-12-12T01:42:31.785 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2023-12-12T01:42:31.787 INFO:tasks.ceph.cluster1.mgr.x:Started 2023-12-12T01:42:31.787 INFO:tasks.ceph:Setting crush tunables to default 2023-12-12T01:42:31.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd crush tunables default 2023-12-12T01:42:32.058 INFO:teuthology.orchestra.run.smithi055.stderr:adjusted tunables profile to default 2023-12-12T01:42:32.077 INFO:tasks.ceph:check_enable_crimson: False 2023-12-12T01:42:32.077 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2023-12-12T01:42:32.077 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:32.077 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2023-12-12T01:42:32.090 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:32.091 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2023-12-12T01:42:32.146 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:32.147 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2023-12-12T01:42:32.203 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd new 9f50009f-ef20-4a77-9de9-cd9707279076 0 2023-12-12T01:42:32.322 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:32.321+0000 7f75d10a1180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-12T01:42:32.458 INFO:teuthology.orchestra.run.smithi055.stdout:0 2023-12-12T01:42:32.475 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd new 4ac775af-1ed3-4717-9081-30a58cc61291 1 2023-12-12T01:42:32.674 INFO:teuthology.orchestra.run.smithi055.stdout:1 2023-12-12T01:42:32.692 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd new e7149351-6dc2-4e78-83d3-e37f097e7d60 2 2023-12-12T01:42:32.897 INFO:teuthology.orchestra.run.smithi055.stdout:2 2023-12-12T01:42:32.914 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2023-12-12T01:42:32.915 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2023-12-12T01:42:32.917 INFO:tasks.ceph.cluster1.osd.0:Started 2023-12-12T01:42:32.918 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2023-12-12T01:42:32.918 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2023-12-12T01:42:32.920 INFO:tasks.ceph.cluster1.osd.1:Started 2023-12-12T01:42:32.921 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2023-12-12T01:42:32.921 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2023-12-12T01:42:32.923 INFO:tasks.ceph.cluster1.osd.2:Started 2023-12-12T01:42:32.923 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:33.025 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:33.025+0000 7f75d10a1180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-12T01:42:33.219 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:33.221+0000 7f75d10a1180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-12T01:42:33.322 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:33.322 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":5,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:32.893837+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:33.339 INFO:tasks.ceph.ceph_manager.cluster1:[] 2023-12-12T01:42:33.339 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-12T01:42:33.544 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T01:42:33.545+0000 7f214aaac640 -1 Falling back to public interface 2023-12-12T01:42:33.560 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T01:42:33.561+0000 7f2a19fee640 -1 Falling back to public interface 2023-12-12T01:42:33.564 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:42:33.565+0000 7f97cb53a640 -1 Falling back to public interface 2023-12-12T01:42:33.642 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:33.824 INFO:teuthology.misc.health.smithi055.stdout: 2023-12-12T01:42:33.824 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":5,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:32.893837+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:33.839 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-12T01:42:33.843 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:33.845+0000 7f75d10a1180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-12T01:42:33.900 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:33.901+0000 7f75d10a1180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-12T01:42:33.954 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:33.953+0000 7f75d10a1180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-12T01:42:34.034 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.033+0000 7f75d10a1180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-12T01:42:34.091 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.089+0000 7f75d10a1180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-12T01:42:34.168 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.169+0000 7f75d10a1180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-12T01:42:34.223 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.225+0000 7f75d10a1180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-12T01:42:34.368 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.369+0000 7f75d10a1180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-12T01:42:34.430 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:34.429+0000 7f75d10a1180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-12T01:42:35.046 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-12T01:42:35.046 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-12T01:42:35.046 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr: from numpy import show_config as show_numpy_config 2023-12-12T01:42:35.089 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:35.089+0000 7f75d10a1180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-12T01:42:35.220 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:35.221+0000 7f75d10a1180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-12T01:42:35.377 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:35.377+0000 7f75d10a1180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-12T01:42:35.443 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:35.445+0000 7f75d10a1180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-12T01:42:35.572 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:35.573+0000 7f75d10a1180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-12T01:42:36.201 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.201+0000 7f75d10a1180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-12T01:42:36.279 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.281+0000 7f75d10a1180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-12T01:42:36.493 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.493+0000 7f75d10a1180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-12T01:42:36.575 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.577+0000 7f75d10a1180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-12T01:42:36.891 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.893+0000 7f75d10a1180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-12T01:42:36.953 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:36.953+0000 7f75d10a1180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-12T01:42:37.163 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T01:42:37.165+0000 7f214aaac640 -1 osd.0 0 log_to_monitors true 2023-12-12T01:42:37.186 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:42:37.185+0000 7f97cb53a640 -1 osd.1 0 log_to_monitors true 2023-12-12T01:42:37.493 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T01:42:37.493+0000 7f2a19fee640 -1 osd.2 0 log_to_monitors true 2023-12-12T01:42:37.993 INFO:tasks.ceph.cluster1.mgr.x.smithi055.stderr:2023-12-12T01:42:37.993+0000 7f75787e9640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-12T01:42:40.142 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:40.601 INFO:teuthology.misc.health.smithi055.stdout: 2023-12-12T01:42:40.602 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":7,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:39.054128+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:40.630 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-12T01:42:41.140 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:42:41.141+0000 7f97c84f6640 -1 osd.1 0 waiting for initial osdmap 2023-12-12T01:42:41.172 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:42:41.173+0000 7f97bf30d640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:42:41.195 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T01:42:41.197+0000 7f2a15774640 -1 osd.2 0 waiting for initial osdmap 2023-12-12T01:42:41.217 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T01:42:41.217+0000 7f2146232640 -1 osd.0 0 waiting for initial osdmap 2023-12-12T01:42:41.225 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T01:42:41.225+0000 7f2a0ddc1640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:42:41.248 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T01:42:41.249+0000 7f213e87f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:42:46.935 DEBUG:teuthology.orchestra.run.smithi055:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:47.236 INFO:teuthology.misc.health.smithi055.stdout: 2023-12-12T01:42:47.237 INFO:teuthology.misc.health.smithi055.stdout:{"epoch":11,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:45.150703+0000","last_up_change":"2023-12-12T01:42:42.145792+0000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:42:43.067634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6801","nonce":218205279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6803","nonce":218205279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6807","nonce":218205279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6805","nonce":218205279}]},"public_addr":"172.21.15.55:6801/218205279","cluster_addr":"172.21.15.55:6803/218205279","heartbeat_back_addr":"172.21.15.55:6807/218205279","heartbeat_front_addr":"172.21.15.55:6805/218205279","state":["exists","up"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6817","nonce":624247608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6819","nonce":624247608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6823","nonce":624247608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6821","nonce":624247608}]},"public_addr":"172.21.15.55:6817/624247608","cluster_addr":"172.21.15.55:6819/624247608","heartbeat_back_addr":"172.21.15.55:6823/624247608","heartbeat_front_addr":"172.21.15.55:6821/624247608","state":["exists","up"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6809","nonce":859848655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6811","nonce":859848655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6815","nonce":859848655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6813","nonce":859848655}]},"public_addr":"172.21.15.55:6809/859848655","cluster_addr":"172.21.15.55:6811/859848655","heartbeat_back_addr":"172.21.15.55:6815/859848655","heartbeat_front_addr":"172.21.15.55:6813/859848655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.157627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.162269+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.509259+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:47.254 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-12T01:42:47.254 INFO:tasks.ceph:Creating RBD pool 2023-12-12T01:42:47.254 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd pool create rbd 8 2023-12-12T01:42:47.879 INFO:teuthology.orchestra.run.smithi055.stderr:pool 'rbd' created 2023-12-12T01:42:47.906 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-12T01:42:48.881 INFO:teuthology.orchestra.run.smithi055.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-12T01:42:48.910 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2023-12-12T01:42:48.911 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2023-12-12T01:42:48.911 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2023-12-12T01:42:48.911 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2023-12-12T01:42:48.912 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-12T01:42:49.257 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:49.274 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6824","nonce":217747385},{"type":"v1","addr":"172.21.15.55:6825","nonce":217747385}]},"active_addr":"172.21.15.55:6825/217747385","active_change":"2023-12-12T01:42:36.976031+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1928885209}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":1704358001}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":3122125625}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2330489875}]}]} 2023-12-12T01:42:49.277 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2023-12-12T01:42:49.278 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2023-12-12T01:42:49.278 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:49.573 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:49.573 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":13,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:48.882350+0000","last_up_change":"2023-12-12T01:42:42.145792+0000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:42:43.067634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-12T01:42:47.590187+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6801","nonce":218205279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6803","nonce":218205279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6807","nonce":218205279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6805","nonce":218205279}]},"public_addr":"172.21.15.55:6801/218205279","cluster_addr":"172.21.15.55:6803/218205279","heartbeat_back_addr":"172.21.15.55:6807/218205279","heartbeat_front_addr":"172.21.15.55:6805/218205279","state":["exists","up"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6817","nonce":624247608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6819","nonce":624247608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6823","nonce":624247608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6821","nonce":624247608}]},"public_addr":"172.21.15.55:6817/624247608","cluster_addr":"172.21.15.55:6819/624247608","heartbeat_back_addr":"172.21.15.55:6823/624247608","heartbeat_front_addr":"172.21.15.55:6821/624247608","state":["exists","up"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6809","nonce":859848655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6811","nonce":859848655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6815","nonce":859848655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6813","nonce":859848655}]},"public_addr":"172.21.15.55:6809/859848655","cluster_addr":"172.21.15.55:6811/859848655","heartbeat_back_addr":"172.21.15.55:6815/859848655","heartbeat_front_addr":"172.21.15.55:6813/859848655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.157627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.162269+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.509259+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:49.592 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2023-12-12T01:42:49.592 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-12T01:42:49.875 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:49.876 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":13,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T01:42:48.882350+0000","last_up_change":"2023-12-12T01:42:42.145792+0000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:42:43.067634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-12T01:42:47.590187+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6801","nonce":218205279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6803","nonce":218205279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6807","nonce":218205279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6805","nonce":218205279}]},"public_addr":"172.21.15.55:6801/218205279","cluster_addr":"172.21.15.55:6803/218205279","heartbeat_back_addr":"172.21.15.55:6807/218205279","heartbeat_front_addr":"172.21.15.55:6805/218205279","state":["exists","up"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6817","nonce":624247608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6819","nonce":624247608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6823","nonce":624247608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6821","nonce":624247608}]},"public_addr":"172.21.15.55:6817/624247608","cluster_addr":"172.21.15.55:6819/624247608","heartbeat_back_addr":"172.21.15.55:6823/624247608","heartbeat_front_addr":"172.21.15.55:6821/624247608","state":["exists","up"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6809","nonce":859848655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6811","nonce":859848655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6815","nonce":859848655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6813","nonce":859848655}]},"public_addr":"172.21.15.55:6809/859848655","cluster_addr":"172.21.15.55:6811/859848655","heartbeat_back_addr":"172.21.15.55:6815/859848655","heartbeat_front_addr":"172.21.15.55:6813/859848655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.157627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.162269+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.509259+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:42:49.894 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2023-12-12T01:42:49.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2023-12-12T01:42:49.895 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2023-12-12T01:42:50.045 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:50.046 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-12T01:42:50.048 INFO:teuthology.orchestra.run.smithi055.stdout:34359738373 2023-12-12T01:42:50.048 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-12T01:42:50.249 INFO:teuthology.orchestra.run.smithi055.stdout:34359738373 2023-12-12T01:42:50.249 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-12T01:42:50.372 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:50.373 INFO:teuthology.orchestra.run.smithi055.stdout:34359738371 2023-12-12T01:42:50.392 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-12-12T01:42:50.393 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2023-12-12T01:42:50.523 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:50.540 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-12-12T01:42:51.392 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-12T01:42:51.393 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-12T01:42:51.541 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-12T01:42:51.692 INFO:teuthology.orchestra.run.smithi055.stdout:34359738371 2023-12-12T01:42:51.702 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:51.710 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738371 for osd.1 2023-12-12T01:42:51.720 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2023-12-12T01:42:51.817 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:51.834 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2023-12-12T01:42:52.711 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2023-12-12T01:42:52.722 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2023-12-12T01:42:52.835 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2023-12-12T01:42:53.078 INFO:teuthology.orchestra.run.smithi055.stdout:34359738372 2023-12-12T01:42:53.095 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.1 2023-12-12T01:42:53.096 DEBUG:teuthology.parallel:result is None 2023-12-12T01:42:53.098 INFO:teuthology.orchestra.run.smithi055.stdout:34359738373 2023-12-12T01:42:53.115 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2023-12-12T01:42:53.116 DEBUG:teuthology.parallel:result is None 2023-12-12T01:42:53.120 INFO:teuthology.orchestra.run.smithi055.stdout:34359738373 2023-12-12T01:42:53.138 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2023-12-12T01:42:53.138 DEBUG:teuthology.parallel:result is None 2023-12-12T01:42:53.138 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-12T01:42:53.138 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T01:42:53.468 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:53.468 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T01:42:53.486 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-12T01:42:52.991937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95040,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020480,"statfs":{"total":289910292480,"available":289812971520,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105358"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.902776+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T01:42:48.902776+0000","last_peered":"2023-12-12T01:42:48.902776+0000","last_clean":"2023-12-12T01:42:48.902776+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T01:42:48.902776+0000","last_undegraded":"2023-12-12T01:42:48.902776+0000","last_fullsized":"2023-12-12T01:42:48.902776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:13:13.826038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.911170+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T01:42:48.911170+0000","last_peered":"2023-12-12T01:42:48.911170+0000","last_clean":"2023-12-12T01:42:48.911170+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T01:42:48.911170+0000","last_undegraded":"2023-12-12T01:42:48.911170+0000","last_fullsized":"2023-12-12T01:42:48.911170+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:07:52.177818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.911476+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T01:42:48.911476+0000","last_peered":"2023-12-12T01:42:48.911476+0000","last_clean":"2023-12-12T01:42:48.911476+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T01:42:48.911476+0000","last_undegraded":"2023-12-12T01:42:48.911476+0000","last_fullsized":"2023-12-12T01:42:48.911476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:38:15.007896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.908778+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T01:42:48.908778+0000","last_peered":"2023-12-12T01:42:48.908778+0000","last_clean":"2023-12-12T01:42:48.908778+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T01:42:48.908778+0000","last_undegraded":"2023-12-12T01:42:48.908778+0000","last_fullsized":"2023-12-12T01:42:48.908778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:26:19.849073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.909922+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T01:42:48.909922+0000","last_peered":"2023-12-12T01:42:48.909922+0000","last_clean":"2023-12-12T01:42:48.909922+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T01:42:48.909922+0000","last_undegraded":"2023-12-12T01:42:48.909922+0000","last_fullsized":"2023-12-12T01:42:48.909922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:40:38.435989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:42:49.898272+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T01:42:49.898272+0000","last_peered":"2023-12-12T01:42:49.898272+0000","last_clean":"2023-12-12T01:42:49.898272+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T01:42:49.898272+0000","last_undegraded":"2023-12-12T01:42:49.898272+0000","last_fullsized":"2023-12-12T01:42:49.898272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:42:49.898231+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T01:42:49.898231+0000","last_peered":"2023-12-12T01:42:49.898231+0000","last_clean":"2023-12-12T01:42:49.898231+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T01:42:49.898231+0000","last_undegraded":"2023-12-12T01:42:49.898231+0000","last_fullsized":"2023-12-12T01:42:49.898231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.905755+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T01:42:48.905755+0000","last_peered":"2023-12-12T01:42:48.905755+0000","last_clean":"2023-12-12T01:42:48.905755+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T01:42:48.905755+0000","last_undegraded":"2023-12-12T01:42:48.905755+0000","last_fullsized":"2023-12-12T01:42:48.905755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:05.363458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.068247+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T01:42:48.068247+0000","last_peered":"2023-12-12T01:42:48.068247+0000","last_clean":"2023-12-12T01:42:48.068247+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T01:42:48.068247+0000","last_undegraded":"2023-12-12T01:42:48.068247+0000","last_fullsized":"2023-12-12T01:42:48.068247+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T01:42:53.487 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T01:42:53.775 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:53.775 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T01:42:53.793 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-12T01:42:52.991937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":95040,"kb_used_data":1496,"kb_used_omap":4,"kb_used_meta":93499,"kb_avail":283020480,"statfs":{"total":289910292480,"available":289812971520,"internally_reserved":0,"allocated":1531904,"data_stored":1274729,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":95743311},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.105358"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.902776+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T01:42:48.902776+0000","last_peered":"2023-12-12T01:42:48.902776+0000","last_clean":"2023-12-12T01:42:48.902776+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T01:42:48.902776+0000","last_undegraded":"2023-12-12T01:42:48.902776+0000","last_fullsized":"2023-12-12T01:42:48.902776+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:13:13.826038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.911170+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T01:42:48.911170+0000","last_peered":"2023-12-12T01:42:48.911170+0000","last_clean":"2023-12-12T01:42:48.911170+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T01:42:48.911170+0000","last_undegraded":"2023-12-12T01:42:48.911170+0000","last_fullsized":"2023-12-12T01:42:48.911170+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:07:52.177818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.911476+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T01:42:48.911476+0000","last_peered":"2023-12-12T01:42:48.911476+0000","last_clean":"2023-12-12T01:42:48.911476+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T01:42:48.911476+0000","last_undegraded":"2023-12-12T01:42:48.911476+0000","last_fullsized":"2023-12-12T01:42:48.911476+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:38:15.007896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.908778+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T01:42:48.908778+0000","last_peered":"2023-12-12T01:42:48.908778+0000","last_clean":"2023-12-12T01:42:48.908778+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T01:42:48.908778+0000","last_undegraded":"2023-12-12T01:42:48.908778+0000","last_fullsized":"2023-12-12T01:42:48.908778+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:26:19.849073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.909922+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T01:42:48.909922+0000","last_peered":"2023-12-12T01:42:48.909922+0000","last_clean":"2023-12-12T01:42:48.909922+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T01:42:48.909922+0000","last_undegraded":"2023-12-12T01:42:48.909922+0000","last_fullsized":"2023-12-12T01:42:48.909922+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:40:38.435989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:42:49.898272+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T01:42:49.898272+0000","last_peered":"2023-12-12T01:42:49.898272+0000","last_clean":"2023-12-12T01:42:49.898272+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T01:42:49.898272+0000","last_undegraded":"2023-12-12T01:42:49.898272+0000","last_fullsized":"2023-12-12T01:42:49.898272+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:42:49.898231+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T01:42:49.898231+0000","last_peered":"2023-12-12T01:42:49.898231+0000","last_clean":"2023-12-12T01:42:49.898231+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T01:42:49.898231+0000","last_undegraded":"2023-12-12T01:42:49.898231+0000","last_fullsized":"2023-12-12T01:42:49.898231+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":13,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.905755+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T01:42:48.905755+0000","last_peered":"2023-12-12T01:42:48.905755+0000","last_clean":"2023-12-12T01:42:48.905755+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T01:42:48.905755+0000","last_undegraded":"2023-12-12T01:42:48.905755+0000","last_fullsized":"2023-12-12T01:42:48.905755+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:05.363458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":140,"reported_epoch":12,"state":"active+clean","last_fresh":"2023-12-12T01:42:48.068247+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T01:42:48.068247+0000","last_peered":"2023-12-12T01:42:48.068247+0000","last_clean":"2023-12-12T01:42:48.068247+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T01:42:48.068247+0000","last_undegraded":"2023-12-12T01:42:48.068247+0000","last_fullsized":"2023-12-12T01:42:48.068247+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":114688,"data_stored":31331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.628}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84499999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31872,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339968,"statfs":{"total":96636764160,"available":96604127232,"internally_reserved":0,"allocated":708608,"data_stored":621699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T01:42:53.794 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-12T01:42:53.794 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2023-12-12T01:42:53.794 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2023-12-12T01:42:53.795 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2023-12-12T01:42:54.120 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:54.120 INFO:teuthology.orchestra.run.smithi055.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-12T01:42:54.138 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2023-12-12T01:42:54.138 INFO:teuthology.run_tasks:Running task ceph... 2023-12-12T01:42:54.148 INFO:tasks.ceph:Creating ceph cluster cluster2... 2023-12-12T01:42:54.148 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-12-12T01:42:54.149 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'branch': 'wip-dis-testing-1', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum}', 'email': 'idryomov@gmail.com', 'first_in_suite': False, 'job_id': '7488100', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-dis-testing-1'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dis-testing-2', 'sha1': 'e7f1b79d320b95939278401b2a5bd1b491f81fe3'}}, 'owner': 'scheduled_dis@teuthology', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7033, 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef', 'sleep_before_teardown': 0, 'subset': '91/99', 'suite': 'rbd', 'suite_branch': 'wip-dis-testing-2', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci', 'suite_sha1': 'e7f1b79d320b95939278401b2a5bd1b491f81fe3', 'targets': {'smithi055.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhDAJKfrLVA+K4Ad6CRbRTxf9G11RQdKVLaVWOow2atgabyCvvM6RPcIaCoqkzXKn0+HB/ochBaexdCmpXV9EQ=', 'smithi140.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRz2LD1q4ZzlkncmnwebNaW2K130qCITj6zPV4RfGgzvT1s4SG2i00T3UnQlVQKmsySsbKYfD5Kvf2GQXBRRzs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': '3d2db44fa991bc355074aa9b7647f63b77201fef'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73', 'timestamp': '2023-12-12_01:13:29', 'tube': 'smithi', 'user': 'dis', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.232157'} 2023-12-12T01:42:54.149 DEBUG:teuthology.orchestra.run.smithi055:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-12T01:42:54.152 DEBUG:teuthology.orchestra.run.smithi140:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2023-12-12T01:42:54.158 DEBUG:teuthology.orchestra.run.smithi055:> sudo install -d -m0777 -- /var/run/ceph 2023-12-12T01:42:54.200 DEBUG:teuthology.orchestra.run.smithi140:> sudo install -d -m0777 -- /var/run/ceph 2023-12-12T01:42:54.214 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:54.214 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/scratch_devs of=/dev/stdout 2023-12-12T01:42:54.259 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-12-12T01:42:54.259 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_1 2023-12-12T01:42:54.307 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-12-12T01:42:54.307 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 882 Links: 1 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-12-12 01:36:47.510166585 +0000 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-12-12 01:36:47.506166621 +0000 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-12-12 01:36:47.506166621 +0000 2023-12-12T01:42:54.308 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-12-12T01:42:54.308 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-12-12T01:42:54.361 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-12-12T01:42:54.362 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-12-12T01:42:54.362 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000376406 s, 1.4 MB/s 2023-12-12T01:42:54.363 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-12-12T01:42:54.412 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_2 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 901 Links: 1 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-12-12 01:36:48.466158046 +0000 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-12-12 01:36:48.094161368 +0000 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-12-12 01:36:48.094161368 +0000 2023-12-12T01:42:54.460 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-12-12T01:42:54.461 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-12-12T01:42:54.517 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-12-12T01:42:54.517 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-12-12T01:42:54.518 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000344984 s, 1.5 MB/s 2023-12-12T01:42:54.519 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-12-12T01:42:54.569 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_3 2023-12-12T01:42:54.615 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 919 Links: 1 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-12-12 01:36:49.070152651 +0000 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-12-12 01:36:48.678156153 +0000 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-12-12 01:36:48.678156153 +0000 2023-12-12T01:42:54.616 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-12-12T01:42:54.616 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-12-12T01:42:54.669 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-12-12T01:42:54.669 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-12-12T01:42:54.669 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000376031 s, 1.4 MB/s 2023-12-12T01:42:54.671 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-12-12T01:42:54.721 DEBUG:teuthology.orchestra.run.smithi140:> stat /dev/vg_nvme/lv_4 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout:Device: 5h/5d Inode: 932 Links: 1 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout:Access: 2023-12-12 01:36:49.678147220 +0000 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout:Modify: 2023-12-12 01:36:49.294150650 +0000 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout:Change: 2023-12-12 01:36:49.294150650 +0000 2023-12-12T01:42:54.768 INFO:teuthology.orchestra.run.smithi140.stdout: Birth: - 2023-12-12T01:42:54.769 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-12-12T01:42:54.821 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records in 2023-12-12T01:42:54.821 INFO:teuthology.orchestra.run.smithi140.stderr:1+0 records out 2023-12-12T01:42:54.821 INFO:teuthology.orchestra.run.smithi140.stderr:512 bytes copied, 0.000280403 s, 1.8 MB/s 2023-12-12T01:42:54.822 DEBUG:teuthology.orchestra.run.smithi140:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-12-12T01:42:54.873 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:54.873 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi140.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-12T01:42:54.873 INFO:tasks.ceph:Generating config... 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client] debug rbd = 20 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client] log to stderr = False 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client] rbd default features = 125 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:54.874 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:54.875 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:54.875 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-12-12T01:42:54.875 INFO:tasks.ceph:[global] mon warn on pool no app = False 2023-12-12T01:42:54.875 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2023-12-12T01:42:54.875 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-12-12T01:42:54.875 INFO:tasks.ceph:[mgr] debug ms = 1 2023-12-12T01:42:54.875 INFO:tasks.ceph:[mon] debug mon = 20 2023-12-12T01:42:54.875 INFO:tasks.ceph:[mon] debug ms = 1 2023-12-12T01:42:54.875 INFO:tasks.ceph:[mon] debug paxos = 20 2023-12-12T01:42:54.875 INFO:tasks.ceph:[osd] bdev async discard = True 2023-12-12T01:42:54.875 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] debug ms = 1 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] debug osd = 20 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-12-12T01:42:54.876 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-12-12T01:42:54.877 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-12-12T01:42:54.877 INFO:tasks.ceph:Setting up cluster2.mon.a... 2023-12-12T01:42:54.877 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2023-12-12T01:42:54.945 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.keyring 2023-12-12T01:42:54.947 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2023-12-12T01:42:55.023 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-12T01:42:55.078 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.140')] 2023-12-12T01:42:55.078 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.140', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2023-12-12T01:42:55.079 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:55.079 DEBUG:teuthology.orchestra.run.smithi140:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-12-12T01:42:55.123 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.140 --print /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: generated fsid 59e1e678-6e7f-4060-9001-c278538e473e 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:setting min_mon_release = pacific 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:epoch 0 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:fsid 59e1e678-6e7f-4060-9001-c278538e473e 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:last_changed 2023-12-12T01:42:55.205702+0000 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:created 2023-12-12T01:42:55.205702+0000 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:min_mon_release 16 (pacific) 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:election_strategy: 1 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:0: [v2:172.21.15.140:3300/0,v1:172.21.15.140:6789/0] mon.a 2023-12-12T01:42:55.202 INFO:teuthology.orchestra.run.smithi140.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2023-12-12T01:42:55.204 DEBUG:teuthology.orchestra.run.smithi140:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-12-12T01:42:55.252 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 59e1e678-6e7f-4060-9001-c278538e473e... 2023-12-12T01:42:55.253 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-12T01:42:55.256 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout:[global] 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: chdir = "" 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: auth supported = cephx 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: filestore xattr use omap = true 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: mon clock drift allowed = 1.000 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: osd crush chooseleaf type = 0 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: auth debug = true 2023-12-12T01:42:55.298 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on old message = true 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: ms die on bug = true 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon pg warn max object skew = 0 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: # disable pg_autoscaler by default for new pools 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default size = 2 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow primary affinity = true 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd allow pg remap = true 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on legacy crush tunables = false 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on crush straw calc version zero = false 2023-12-12T01:42:55.299 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on no sortbitwise = false 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on osd down out interval zero = false 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on too few osds = false 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon_allow_pool_size_one = true 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: osd default data pool replay window = 5 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.300 INFO:teuthology.orchestra.run.smithi055.stdout: mon allow pool delete = true 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon cluster log file level = debug 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: debug asserts on shutdown = true 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon health detail to clog = false 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon host = 172.21.15.140 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon client directed command retry = 5 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: mon warn on pool no app = False 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: ms inject socket failures = 5000 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: fsid = 59e1e678-6e7f-4060-9001-c278538e473e 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout:[osd] 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: osd journal size = 100 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub load threshold = 5.0 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: osd scrub max interval = 600 2023-12-12T01:42:55.301 INFO:teuthology.orchestra.run.smithi055.stdout: osd mclock profile = high_recovery_ops 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd recover clone overlap = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd recovery max chunk = 1048576 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug shutdown = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug op order = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug verify stray on activate = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug trim objects = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd open classes on start = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug pg log writeout = true 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.302 INFO:teuthology.orchestra.run.smithi055.stdout: osd deep scrub update digest min age = 30 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: osd map max advance = 10 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: journal zero on create = true 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: filestore ondisk finisher threads = 3 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: filestore apply finisher threads = 3 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.303 INFO:teuthology.orchestra.run.smithi055.stdout: bdev debug aio = true 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: osd debug misdirected ops = true 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: bdev async discard = True 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: bdev enable discard = True 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore allocator = stupid 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore block size = 96636764160 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: bluestore fsck on mount = True 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluefs = 1/20 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug bluestore = 1/20 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug osd = 20 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: debug rocksdb = 4/10 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd full ratio = 0.9 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd nearfull ratio = 0.8 2023-12-12T01:42:55.304 INFO:teuthology.orchestra.run.smithi055.stdout: osd failsafe full ratio = 0.95 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: osd objectstore = bluestore 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout:[mgr] 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: debug mgr = 20 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: mgr/telemetry/nag = false 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.305 INFO:teuthology.orchestra.run.smithi055.stdout:[mon] 2023-12-12T01:42:55.306 INFO:teuthology.orchestra.run.smithi055.stdout: debug ms = 1 2023-12-12T01:42:55.306 INFO:teuthology.orchestra.run.smithi055.stdout: debug mon = 20 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: debug paxos = 20 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: debug auth = 20 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon data avail warn = 5 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon mgr mkfs grace = 240 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd reporter subtree level = osd 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon osd prime pg temp = true 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: auth mon ticket ttl = 660 # 11m 2023-12-12T01:42:55.307 INFO:teuthology.orchestra.run.smithi055.stdout: auth service ticket ttl = 240 # 4m 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: # don't complain about insecure global_id in the test suite 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: # 1m isn't quite enough 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: mon_down_mkfs_grace = 2m 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: mon_warn_on_filestore_osds = false 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout:[client] 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: rgw cache enabled = true 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable ops log = true 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: rgw enable usage log = true 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-12T01:42:55.308 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: debug rbd = 20 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: debug rbd_mirror = 15 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: log to stderr = False 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: rbd default features = 125 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: rbd mirror image policy type = simple 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:[mon.a] 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.0] 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.1] 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.2] 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.309 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.310 INFO:teuthology.orchestra.run.smithi055.stdout:[client.mirror.3] 2023-12-12T01:42:55.310 INFO:teuthology.orchestra.run.smithi055.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.310 INFO:teuthology.orchestra.run.smithi055.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.323 INFO:teuthology.orchestra.run.smithi140.stdout:[global] 2023-12-12T01:42:55.323 INFO:teuthology.orchestra.run.smithi140.stdout: chdir = "" 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: auth supported = cephx 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: filestore xattr use omap = true 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: mon clock drift allowed = 1.000 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: osd crush chooseleaf type = 0 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: auth debug = true 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on old message = true 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: ms die on bug = true 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.324 INFO:teuthology.orchestra.run.smithi140.stdout: mon max pg per osd = 10000 # >= luminous 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon pg warn max object skew = 0 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: # disable pg_autoscaler by default for new pools 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: osd_pool_default_pg_autoscale_mode = off 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default size = 2 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow primary affinity = true 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd allow pg remap = true 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on legacy crush tunables = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on crush straw calc version zero = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on no sortbitwise = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on osd down out interval zero = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on too few osds = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-12-12T01:42:55.325 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_pool_no_redundancy = false 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon_allow_pool_size_one = true 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: osd default data pool replay window = 5 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon allow pool delete = true 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon cluster log file level = debug 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: debug asserts on shutdown = true 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon health detail to clog = false 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon host = 172.21.15.140 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon client directed command retry = 5 2023-12-12T01:42:55.326 INFO:teuthology.orchestra.run.smithi140.stdout: mon warn on pool no app = False 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: ms inject socket failures = 5000 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: fsid = 59e1e678-6e7f-4060-9001-c278538e473e 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout:[osd] 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd journal size = 100 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub load threshold = 5.0 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd scrub max interval = 600 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd mclock profile = high_recovery_ops 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd recover clone overlap = true 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd recovery max chunk = 1048576 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug shutdown = true 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug op order = true 2023-12-12T01:42:55.327 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug verify stray on activate = true 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug trim objects = true 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: osd open classes on start = true 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug pg log writeout = true 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: osd deep scrub update digest min age = 30 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: osd map max advance = 10 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: journal zero on create = true 2023-12-12T01:42:55.328 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: filestore ondisk finisher threads = 3 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: filestore apply finisher threads = 3 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bdev debug aio = true 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: osd debug misdirected ops = true 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bdev async discard = True 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bdev enable discard = True 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore allocator = stupid 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore block size = 96636764160 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: bluestore fsck on mount = True 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluefs = 1/20 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: debug bluestore = 1/20 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:55.329 INFO:teuthology.orchestra.run.smithi140.stdout: debug osd = 20 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: debug rocksdb = 4/10 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd backfillfull_ratio = 0.85 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd full ratio = 0.9 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd nearfull ratio = 0.8 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: osd failsafe full ratio = 0.95 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: osd objectstore = bluestore 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout:[mgr] 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: debug mgr = 20 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:55.330 INFO:teuthology.orchestra.run.smithi140.stdout: mgr/telemetry/nag = false 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout:[mon] 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: debug ms = 1 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: debug mon = 20 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: debug paxos = 20 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: debug auth = 20 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon data avail warn = 5 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon mgr mkfs grace = 240 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min pgs per osd = 4 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd reporter subtree level = osd 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon osd prime pg temp = true 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: mon reweight min bytes per osd = 10 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: auth mon ticket ttl = 660 # 11m 2023-12-12T01:42:55.331 INFO:teuthology.orchestra.run.smithi140.stdout: auth service ticket ttl = 240 # 4m 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: # don't complain about insecure global_id in the test suite 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: # 1m isn't quite enough 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: mon_down_mkfs_grace = 2m 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: mon_warn_on_filestore_osds = false 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout:[client] 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: rgw cache enabled = true 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable ops log = true 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: rgw enable usage log = true 2023-12-12T01:42:55.332 INFO:teuthology.orchestra.run.smithi140.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: debug rbd = 20 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: debug rbd_mirror = 15 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: log to stderr = False 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: rbd default features = 125 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: rbd mirror image policy type = simple 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout:[mon.a] 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.0] 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.1] 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.2] 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.333 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.334 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.3] 2023-12-12T01:42:55.334 INFO:teuthology.orchestra.run.smithi140.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2023-12-12T01:42:55.334 INFO:teuthology.orchestra.run.smithi140.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2023-12-12T01:42:55.335 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2023-12-12T01:42:55.335 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2023-12-12T01:42:55.426 INFO:tasks.ceph:Copying monmap to all nodes... 2023-12-12T01:42:55.426 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:55.426 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2023-12-12T01:42:55.476 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:55.476 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2023-12-12T01:42:55.523 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T01:42:55.523 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:55.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-12T01:42:55.523 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-12T01:42:55.549 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:42:55.549 DEBUG:teuthology.orchestra.run.smithi055:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:42:55.596 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:42:55.596 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:55.596 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd of=/etc/ceph/cluster2.keyring 2023-12-12T01:42:55.596 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2023-12-12T01:42:55.618 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:42:55.619 DEBUG:teuthology.orchestra.run.smithi140:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:42:55.671 INFO:tasks.ceph:Setting up mon nodes... 2023-12-12T01:42:55.672 INFO:tasks.ceph:Setting up mgr nodes... 2023-12-12T01:42:55.672 DEBUG:teuthology.orchestra.run.smithi055:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-12T01:42:55.718 INFO:teuthology.orchestra.run.smithi055.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2023-12-12T01:42:55.720 INFO:tasks.ceph:Setting up mds nodes... 2023-12-12T01:42:55.721 INFO:tasks.ceph_client:Setting up client nodes... 2023-12-12T01:42:55.721 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2023-12-12T01:42:55.796 INFO:teuthology.orchestra.run.smithi055.stdout:creating /etc/ceph/cluster2.client.0.keyring 2023-12-12T01:42:55.807 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2023-12-12T01:42:55.842 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2023-12-12T01:42:55.852 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-12T01:42:55.930 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2023-12-12T01:42:55.942 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-12T01:42:56.013 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2023-12-12T01:42:56.022 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-12T01:42:56.097 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2023-12-12T01:42:56.108 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-12T01:42:56.179 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2023-12-12T01:42:56.189 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-12T01:42:56.259 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2023-12-12T01:42:56.269 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-12T01:42:56.339 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2023-12-12T01:42:56.348 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-12T01:42:56.421 INFO:teuthology.orchestra.run.smithi140.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2023-12-12T01:42:56.430 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-12-12T01:42:56.430 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi055.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi140.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2023-12-12T01:42:56.430 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2023-12-12T01:42:56.486 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:56.487 INFO:tasks.ceph:role: cluster2.osd.0 2023-12-12T01:42:56.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:42:56.487 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:56.551 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:57.001 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-12-12T01:42:57.003 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-12-12T01:42:57.003 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2023-12-12T01:42:57.107 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2023-12-12T01:42:57.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:57.117 INFO:teuthology.orchestra.run.smithi140.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:57.117 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2023-12-12T01:42:57.169 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:57.169 INFO:tasks.ceph:role: cluster2.osd.1 2023-12-12T01:42:57.169 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:42:57.170 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-12-12T01:42:57.231 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:57.232 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:57.702 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-12-12T01:42:57.704 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-12-12T01:42:57.704 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2023-12-12T01:42:57.857 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2023-12-12T01:42:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:57.869 INFO:teuthology.orchestra.run.smithi140.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:57.869 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2023-12-12T01:42:57.924 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2023-12-12T01:42:57.924 INFO:tasks.ceph:role: cluster2.osd.2 2023-12-12T01:42:57.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:42:57.924 DEBUG:teuthology.orchestra.run.smithi140:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-12-12T01:42:57.987 INFO:teuthology.orchestra.run.smithi140.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-12-12T01:42:57.987 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 attr=2, projid32bit=1 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout: = sunit=0 swidth=0 blks 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-12-12T01:42:57.988 INFO:teuthology.orchestra.run.smithi140.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-12-12T01:42:58.470 INFO:teuthology.orchestra.run.smithi140.stdout:Discarding blocks...Done. 2023-12-12T01:42:58.472 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi140.front.sepia.ceph.com -o noatime 2023-12-12T01:42:58.472 DEBUG:teuthology.orchestra.run.smithi140:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2023-12-12T01:42:58.617 DEBUG:teuthology.orchestra.run.smithi140:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2023-12-12T01:42:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:42:58.629 INFO:teuthology.orchestra.run.smithi140.stderr:sudo: /sbin/restorecon: command not found 2023-12-12T01:42:58.630 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:42:58.745 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2023-12-12T01:42:58.746 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2023-12-12T01:42:58.746 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-12T01:42:58.746 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-12T01:42:58.746 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 bluestore(/var/lib/ceph/osd/cluster2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-0/block: (2) No such file or directory 2023-12-12T01:42:58.746 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:42:58.744+0000 7f708a856640 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2023-12-12T01:43:01.793 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2023-12-12T01:43:01.808 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:43:01.889 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.888+0000 7f507b11b640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2023-12-12T01:43:01.891 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.888+0000 7f507b11b640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2023-12-12T01:43:01.891 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.888+0000 7f507b11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-12T01:43:01.891 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.888+0000 7f507b11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-12T01:43:01.891 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.888+0000 7f507b11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-1/block: (2) No such file or directory 2023-12-12T01:43:01.891 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:01.892+0000 7f507b11b640 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2023-12-12T01:43:04.907 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2023-12-12T01:43:04.962 DEBUG:teuthology.orchestra.run.smithi140:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:43:05.050 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.048+0000 7f812d137640 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2023-12-12T01:43:05.051 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.048+0000 7f812d137640 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2023-12-12T01:43:05.051 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.048+0000 7f812d137640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-12T01:43:05.051 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.048+0000 7f812d137640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-12T01:43:05.051 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.048+0000 7f812d137640 -1 bluestore(/var/lib/ceph/osd/cluster2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/cluster2-2/block: (2) No such file or directory 2023-12-12T01:43:05.051 INFO:teuthology.orchestra.run.smithi140.stderr:2023-12-12T01:43:05.052+0000 7f812d137640 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2023-12-12T01:43:08.063 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2023-12-12T01:43:08.123 INFO:tasks.ceph:Reading keys from all nodes... 2023-12-12T01:43:08.123 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:43:08.123 DEBUG:teuthology.orchestra.run.smithi055:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2023-12-12T01:43:08.138 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.138 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2023-12-12T01:43:08.178 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.179 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2023-12-12T01:43:08.234 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.234 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2023-12-12T01:43:08.286 DEBUG:teuthology.orchestra.run.smithi055:> set -ex 2023-12-12T01:43:08.287 DEBUG:teuthology.orchestra.run.smithi055:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2023-12-12T01:43:08.293 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.293 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2023-12-12T01:43:08.339 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.339 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2023-12-12T01:43:08.387 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.387 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2023-12-12T01:43:08.435 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.436 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2023-12-12T01:43:08.484 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.484 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2023-12-12T01:43:08.531 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.531 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2023-12-12T01:43:08.579 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.579 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2023-12-12T01:43:08.628 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:08.628 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2023-12-12T01:43:08.675 INFO:tasks.ceph:Adding keys to all mons... 2023-12-12T01:43:08.675 DEBUG:teuthology.orchestra.run.smithi140:> sudo tee -a /etc/ceph/cluster2.keyring 2023-12-12T01:43:08.729 INFO:teuthology.orchestra.run.smithi140.stdout:[mgr.x] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundlUcAEKxAABu9XBnQ8t3e8UmrihRRwOQ== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.0] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCiundluXKmLBAA70Txjnyj5wvXvNOJhUbQYg== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.1] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQClundloLFANRAAFICP++TyF2Q2VXFE2uDJIQ== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[osd.2] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCpundll3c4AxAAs2UNTG3A2N7l5U+nYSiUQQ== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[client.0] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundlFhSxLxAAy/gIY2gn0nrcDVXIhtlbnQ== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror] 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundl1utuMhAAKULIn9Yls8SWlyRCyN4/lQ== 2023-12-12T01:43:08.730 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.0] 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundlI6qqNxAA9JCdyzMWfl98hTjTMRNZzw== 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.1] 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlqHoBARAAlpAZGEnX+xEle30Kx8nj6g== 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.2] 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlb9ILBhAAGNVGMFEJ8lgw8HB/uWqvTw== 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.3] 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlsVflChAAno4G89BpmZxNREpRpEfNGg== 2023-12-12T01:43:08.731 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.4] 2023-12-12T01:43:08.732 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundl5fG0DxAAF12CySx1P4Jg+cwQxsKDTA== 2023-12-12T01:43:08.732 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.5] 2023-12-12T01:43:08.732 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlLop0FBAA4NkwDGAqeOaTegkPq3sFXA== 2023-12-12T01:43:08.732 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.6] 2023-12-12T01:43:08.732 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundl99pNGRAAN54yJE0wQTbW19GZQ+u76A== 2023-12-12T01:43:08.733 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-12-12T01:43:08.819 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:43:08.861 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:43:08.946 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-12-12T01:43:09.032 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.077 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.163 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.255 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.301 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.347 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.390 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.474 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.562 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-12-12T01:43:09.650 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-12-12T01:43:09.650 DEBUG:teuthology.orchestra.run.smithi140:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2023-12-12T01:43:09.662 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2023-12-12T01:43:09.753 DEBUG:teuthology.orchestra.run.smithi140:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2023-12-12T01:43:09.763 DEBUG:teuthology.orchestra.run.smithi140:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2023-12-12T01:43:09.810 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2023-12-12T01:43:09.810 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2023-12-12T01:43:09.810 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2023-12-12T01:43:09.852 INFO:tasks.ceph.cluster2.mon.a:Started 2023-12-12T01:43:09.852 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2023-12-12T01:43:09.853 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2023-12-12T01:43:09.853 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2023-12-12T01:43:09.855 INFO:tasks.ceph.cluster2.mgr.x:Started 2023-12-12T01:43:09.855 INFO:tasks.ceph:Setting crush tunables to default 2023-12-12T01:43:09.855 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd crush tunables default 2023-12-12T01:43:10.114 INFO:teuthology.orchestra.run.smithi140.stderr:adjusted tunables profile to default 2023-12-12T01:43:10.131 INFO:tasks.ceph:check_enable_crimson: False 2023-12-12T01:43:10.131 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2023-12-12T01:43:10.131 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:10.131 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2023-12-12T01:43:10.143 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:10.143 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2023-12-12T01:43:10.197 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:43:10.198 DEBUG:teuthology.orchestra.run.smithi140:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2023-12-12T01:43:10.231 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:10.232+0000 7fb59fa0e180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-12-12T01:43:10.249 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd new 49b51d40-2d01-42ca-a19b-852caf9c0eaf 0 2023-12-12T01:43:10.493 INFO:teuthology.orchestra.run.smithi140.stdout:0 2023-12-12T01:43:10.510 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd new b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c 1 2023-12-12T01:43:10.623 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:10.624+0000 7fb59fa0e180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-12-12T01:43:10.713 INFO:teuthology.orchestra.run.smithi140.stdout:1 2023-12-12T01:43:10.731 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd new bbb61b8a-f28f-428c-8974-af3c28fb4dd9 2 2023-12-12T01:43:10.756 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:10.756+0000 7fb59fa0e180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-12-12T01:43:10.934 INFO:teuthology.orchestra.run.smithi140.stdout:2 2023-12-12T01:43:10.952 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2023-12-12T01:43:10.953 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2023-12-12T01:43:10.954 INFO:tasks.ceph.cluster2.osd.0:Started 2023-12-12T01:43:10.955 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2023-12-12T01:43:10.955 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2023-12-12T01:43:10.957 INFO:tasks.ceph.cluster2.osd.1:Started 2023-12-12T01:43:10.957 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2023-12-12T01:43:10.957 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2023-12-12T01:43:10.959 INFO:tasks.ceph.cluster2.osd.2:Started 2023-12-12T01:43:10.959 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:11.141 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:11.141 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":5,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:10.937248+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:11.158 INFO:tasks.ceph.ceph_manager.cluster2:[] 2023-12-12T01:43:11.158 INFO:tasks.ceph:Waiting for OSDs to come up 2023-12-12T01:43:11.198 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.196+0000 7fb59fa0e180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-12-12T01:43:11.250 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.248+0000 7fb59fa0e180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-12-12T01:43:11.300 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.300+0000 7fb59fa0e180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-12-12T01:43:11.378 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.376+0000 7fb59fa0e180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-12-12T01:43:11.429 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.428+0000 7fb59fa0e180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-12-12T01:43:11.459 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:11.504 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.504+0000 7fb59fa0e180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-12-12T01:43:11.555 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.556+0000 7fb59fa0e180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-12-12T01:43:11.579 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:43:11.580+0000 7fd7bcc07640 -1 Falling back to public interface 2023-12-12T01:43:11.591 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T01:43:11.592+0000 7f5c6276c640 -1 Falling back to public interface 2023-12-12T01:43:11.592 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T01:43:11.592+0000 7f7684c8c640 -1 Falling back to public interface 2023-12-12T01:43:11.650 INFO:teuthology.misc.health.smithi140.stdout: 2023-12-12T01:43:11.651 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":5,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:10.937248+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:11.666 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-12T01:43:11.688 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.688+0000 7fb59fa0e180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-12-12T01:43:11.741 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.740+0000 7fb59fa0e180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-12-12T01:43:11.965 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2023-12-12T01:43:11.965 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2023-12-12T01:43:11.965 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr: from numpy import show_config as show_numpy_config 2023-12-12T01:43:11.972 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:11.972+0000 7fb59fa0e180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-12-12T01:43:12.101 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.100+0000 7fb59fa0e180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-12-12T01:43:12.213 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.212+0000 7fb59fa0e180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-12-12T01:43:12.276 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.276+0000 7fb59fa0e180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-12-12T01:43:12.403 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.404+0000 7fb59fa0e180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-12-12T01:43:12.895 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.896+0000 7fb59fa0e180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-12-12T01:43:12.972 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:12.972+0000 7fb59fa0e180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-12-12T01:43:13.111 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:13.112+0000 7fb59fa0e180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-12-12T01:43:13.186 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:13.188+0000 7fb59fa0e180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-12-12T01:43:13.478 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:13.476+0000 7fb59fa0e180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-12-12T01:43:13.532 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:13.532+0000 7fb59fa0e180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-12-12T01:43:15.324 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:43:15.324+0000 7fd7bcc07640 -1 osd.2 0 log_to_monitors true 2023-12-12T01:43:15.399 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T01:43:15.400+0000 7f7684c8c640 -1 osd.1 0 log_to_monitors true 2023-12-12T01:43:15.411 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T01:43:15.412+0000 7f5c6276c640 -1 osd.0 0 log_to_monitors true 2023-12-12T01:43:16.568 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T01:43:16.568+0000 7fb547159640 -1 mgr.server handle_report got status from non-daemon mon.a 2023-12-12T01:43:17.970 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:18.412 INFO:teuthology.misc.health.smithi140.stdout: 2023-12-12T01:43:18.412 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":7,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:16.949695+0000","last_up_change":"0.000000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:18.438 DEBUG:teuthology.misc:0 of 3 OSDs are up 2023-12-12T01:43:18.911 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T01:43:18.912+0000 7f7681c48640 -1 osd.1 0 waiting for initial osdmap 2023-12-12T01:43:18.938 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:43:18.936+0000 7fd7b9bc3640 -1 osd.2 0 waiting for initial osdmap 2023-12-12T01:43:18.941 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T01:43:18.940+0000 7f7678a5f640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:43:18.975 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:43:18.976+0000 7fd7b09da640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:43:18.988 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T01:43:18.988+0000 7f5c5def2640 -1 osd.0 0 waiting for initial osdmap 2023-12-12T01:43:19.015 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T01:43:19.016+0000 7f5c5653f640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-12-12T01:43:24.742 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:25.034 INFO:teuthology.misc.health.smithi140.stdout: 2023-12-12T01:43:25.035 INFO:teuthology.misc.health.smithi140.stdout:{"epoch":12,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:24.010531+0000","last_up_change":"2023-12-12T01:43:19.964914+0000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:43:21.645718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6809","nonce":125638021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6811","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6813","nonce":125638021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6820","nonce":125638021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6815","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6817","nonce":125638021}]},"public_addr":"172.21.15.140:6809/125638021","cluster_addr":"172.21.15.140:6813/125638021","heartbeat_back_addr":"172.21.15.140:6820/125638021","heartbeat_front_addr":"172.21.15.140:6817/125638021","state":["exists","up"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":10,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6812","nonce":1790311803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6816","nonce":1790311803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6823","nonce":1790311803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6819","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6821","nonce":1790311803}]},"public_addr":"172.21.15.140:6812/1790311803","cluster_addr":"172.21.15.140:6816/1790311803","heartbeat_back_addr":"172.21.15.140:6823/1790311803","heartbeat_front_addr":"172.21.15.140:6821/1790311803","state":["exists","up"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6801","nonce":3638313239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6803","nonce":3638313239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6807","nonce":3638313239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6805","nonce":3638313239}]},"public_addr":"172.21.15.140:6801/3638313239","cluster_addr":"172.21.15.140:6803/3638313239","heartbeat_back_addr":"172.21.15.140:6807/3638313239","heartbeat_front_addr":"172.21.15.140:6805/3638313239","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.450514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.417108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.324939+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:25.052 DEBUG:teuthology.misc:3 of 3 OSDs are up 2023-12-12T01:43:25.052 INFO:tasks.ceph:Creating RBD pool 2023-12-12T01:43:25.053 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd pool create rbd 8 2023-12-12T01:43:26.025 INFO:teuthology.orchestra.run.smithi140.stderr:pool 'rbd' created 2023-12-12T01:43:26.053 DEBUG:teuthology.orchestra.run.smithi140:> sudo ceph --cluster cluster2 osd pool application enable rbd rbd --yes-i-really-mean-it 2023-12-12T01:43:27.027 INFO:teuthology.orchestra.run.smithi140.stderr:enabled application 'rbd' on pool 'rbd' 2023-12-12T01:43:27.059 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2023-12-12T01:43:27.059 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2023-12-12T01:43:27.059 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2023-12-12T01:43:27.060 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2023-12-12T01:43:27.060 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-12-12T01:43:27.388 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:27.405 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":5,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6826","nonce":2994693124},{"type":"v1","addr":"172.21.15.55:6827","nonce":2994693124}]},"active_addr":"172.21.15.55:6827/2994693124","active_change":"2023-12-12T01:43:15.556689+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":3971521199}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":3581870368}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":3550005190}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.55:0","nonce":2477806737}]}]} 2023-12-12T01:43:27.407 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2023-12-12T01:43:27.407 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2023-12-12T01:43:27.408 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:27.695 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:27.695 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":14,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:27.029211+0000","last_up_change":"2023-12-12T01:43:19.964914+0000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:43:21.645718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-12T01:43:25.384237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6809","nonce":125638021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6811","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6813","nonce":125638021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6820","nonce":125638021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6815","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6817","nonce":125638021}]},"public_addr":"172.21.15.140:6809/125638021","cluster_addr":"172.21.15.140:6813/125638021","heartbeat_back_addr":"172.21.15.140:6820/125638021","heartbeat_front_addr":"172.21.15.140:6817/125638021","state":["exists","up"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6812","nonce":1790311803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6816","nonce":1790311803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6823","nonce":1790311803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6819","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6821","nonce":1790311803}]},"public_addr":"172.21.15.140:6812/1790311803","cluster_addr":"172.21.15.140:6816/1790311803","heartbeat_back_addr":"172.21.15.140:6823/1790311803","heartbeat_front_addr":"172.21.15.140:6821/1790311803","state":["exists","up"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6801","nonce":3638313239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6803","nonce":3638313239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6807","nonce":3638313239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6805","nonce":3638313239}]},"public_addr":"172.21.15.140:6801/3638313239","cluster_addr":"172.21.15.140:6803/3638313239","heartbeat_back_addr":"172.21.15.140:6807/3638313239","heartbeat_front_addr":"172.21.15.140:6805/3638313239","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.450514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.417108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.324939+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:27.713 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2023-12-12T01:43:27.713 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2023-12-12T01:43:28.001 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:28.002 INFO:teuthology.orchestra.run.smithi140.stdout:{"epoch":14,"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","created":"2023-12-12T01:43:09.994434+0000","modified":"2023-12-12T01:43:27.029211+0000","last_up_change":"2023-12-12T01:43:19.964914+0000","last_in_change":"2023-12-12T01:43:10.937248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:43:21.645718+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-12T01:43:25.384237+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"49b51d40-2d01-42ca-a19b-852caf9c0eaf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6808","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6809","nonce":125638021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6811","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6813","nonce":125638021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6818","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6820","nonce":125638021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6815","nonce":125638021},{"type":"v1","addr":"172.21.15.140:6817","nonce":125638021}]},"public_addr":"172.21.15.140:6809/125638021","cluster_addr":"172.21.15.140:6813/125638021","heartbeat_back_addr":"172.21.15.140:6820/125638021","heartbeat_front_addr":"172.21.15.140:6817/125638021","state":["exists","up"]},{"osd":1,"uuid":"b2bae4e6-5490-4bc1-afe2-fa6e7ef91b5c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6810","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6812","nonce":1790311803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6814","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6816","nonce":1790311803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6822","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6823","nonce":1790311803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6819","nonce":1790311803},{"type":"v1","addr":"172.21.15.140:6821","nonce":1790311803}]},"public_addr":"172.21.15.140:6812/1790311803","cluster_addr":"172.21.15.140:6816/1790311803","heartbeat_back_addr":"172.21.15.140:6823/1790311803","heartbeat_front_addr":"172.21.15.140:6821/1790311803","state":["exists","up"]},{"osd":2,"uuid":"bbb61b8a-f28f-428c-8974-af3c28fb4dd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6800","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6801","nonce":3638313239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6802","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6803","nonce":3638313239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6806","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6807","nonce":3638313239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.140:6804","nonce":3638313239},{"type":"v1","addr":"172.21.15.140:6805","nonce":3638313239}]},"public_addr":"172.21.15.140:6801/3638313239","cluster_addr":"172.21.15.140:6803/3638313239","heartbeat_back_addr":"172.21.15.140:6807/3638313239","heartbeat_front_addr":"172.21.15.140:6805/3638313239","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.450514+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.417108+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:43:16.324939+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T01:43:28.020 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2023-12-12T01:43:28.020 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2023-12-12T01:43:28.021 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2023-12-12T01:43:28.174 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:28.174 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-12T01:43:28.195 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:28.195 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-12T01:43:28.198 INFO:teuthology.orchestra.run.smithi140.stdout:34359738375 2023-12-12T01:43:28.198 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-12T01:43:28.453 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-12-12T01:43:28.469 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-12T01:43:28.528 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-12-12T01:43:28.533 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:28.548 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-12T01:43:28.552 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.2 2023-12-12T01:43:29.470 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-12T01:43:29.549 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-12T01:43:29.552 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-12T01:43:29.787 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-12-12T01:43:29.805 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.0 2023-12-12T01:43:29.830 INFO:teuthology.orchestra.run.smithi140.stdout:34359738373 2023-12-12T01:43:29.847 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2023-12-12T01:43:29.877 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:29.894 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738374 for osd.2 2023-12-12T01:43:30.806 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2023-12-12T01:43:30.849 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2023-12-12T01:43:30.895 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2023-12-12T01:43:31.118 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:31.134 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.0 2023-12-12T01:43:31.135 DEBUG:teuthology.parallel:result is None 2023-12-12T01:43:31.150 INFO:teuthology.orchestra.run.smithi140.stdout:34359738374 2023-12-12T01:43:31.167 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2023-12-12T01:43:31.167 DEBUG:teuthology.parallel:result is None 2023-12-12T01:43:31.173 INFO:teuthology.orchestra.run.smithi140.stdout:34359738375 2023-12-12T01:43:31.190 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738375 got 34359738375 for osd.2 2023-12-12T01:43:31.190 DEBUG:teuthology.parallel:result is None 2023-12-12T01:43:31.190 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-12T01:43:31.190 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T01:43:31.488 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:31.489 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T01:43:31.505 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-12-12T01:43:29.568400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82968,"kb_used_data":1520,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032552,"statfs":{"total":289910292480,"available":289825333248,"internally_reserved":0,"allocated":1556480,"data_stored":1281932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.536381"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.051825+0000","last_change":"2023-12-12T01:43:27.051716+0000","last_active":"2023-12-12T01:43:28.051825+0000","last_peered":"2023-12-12T01:43:28.051825+0000","last_clean":"2023-12-12T01:43:28.051825+0000","last_became_active":"2023-12-12T01:43:27.051337+0000","last_became_peered":"2023-12-12T01:43:27.051337+0000","last_unstale":"2023-12-12T01:43:28.051825+0000","last_undegraded":"2023-12-12T01:43:28.051825+0000","last_fullsized":"2023-12-12T01:43:28.051825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:49:43.136965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052503+0000","last_change":"2023-12-12T01:43:27.055425+0000","last_active":"2023-12-12T01:43:28.052503+0000","last_peered":"2023-12-12T01:43:28.052503+0000","last_clean":"2023-12-12T01:43:28.052503+0000","last_became_active":"2023-12-12T01:43:27.053612+0000","last_became_peered":"2023-12-12T01:43:27.053612+0000","last_unstale":"2023-12-12T01:43:28.052503+0000","last_undegraded":"2023-12-12T01:43:28.052503+0000","last_fullsized":"2023-12-12T01:43:28.052503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:12:46.742958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052348+0000","last_change":"2023-12-12T01:43:27.053878+0000","last_active":"2023-12-12T01:43:28.052348+0000","last_peered":"2023-12-12T01:43:28.052348+0000","last_clean":"2023-12-12T01:43:28.052348+0000","last_became_active":"2023-12-12T01:43:27.053608+0000","last_became_peered":"2023-12-12T01:43:27.053608+0000","last_unstale":"2023-12-12T01:43:28.052348+0000","last_undegraded":"2023-12-12T01:43:28.052348+0000","last_fullsized":"2023-12-12T01:43:28.052348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T13:09:28.305200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052188+0000","last_change":"2023-12-12T01:43:27.053430+0000","last_active":"2023-12-12T01:43:28.052188+0000","last_peered":"2023-12-12T01:43:28.052188+0000","last_clean":"2023-12-12T01:43:28.052188+0000","last_became_active":"2023-12-12T01:43:27.053063+0000","last_became_peered":"2023-12-12T01:43:27.053063+0000","last_unstale":"2023-12-12T01:43:28.052188+0000","last_undegraded":"2023-12-12T01:43:28.052188+0000","last_fullsized":"2023-12-12T01:43:28.052188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:19:33.765966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:43:27.055069+0000","last_change":"2023-12-12T01:43:27.050662+0000","last_active":"2023-12-12T01:43:27.055069+0000","last_peered":"2023-12-12T01:43:27.055069+0000","last_clean":"2023-12-12T01:43:27.055069+0000","last_became_active":"2023-12-12T01:43:27.049924+0000","last_became_peered":"2023-12-12T01:43:27.049924+0000","last_unstale":"2023-12-12T01:43:27.055069+0000","last_undegraded":"2023-12-12T01:43:27.055069+0000","last_fullsized":"2023-12-12T01:43:27.055069+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:11:29.315127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:43:27.058640+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T01:43:27.058640+0000","last_peered":"2023-12-12T01:43:27.058640+0000","last_clean":"2023-12-12T01:43:27.058640+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T01:43:27.058640+0000","last_undegraded":"2023-12-12T01:43:27.058640+0000","last_fullsized":"2023-12-12T01:43:27.058640+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:59:14.522252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:43:27.058608+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T01:43:27.058608+0000","last_peered":"2023-12-12T01:43:27.058608+0000","last_clean":"2023-12-12T01:43:27.058608+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T01:43:27.058608+0000","last_undegraded":"2023-12-12T01:43:27.058608+0000","last_fullsized":"2023-12-12T01:43:27.058608+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:52.005722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052031+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T01:43:28.052031+0000","last_peered":"2023-12-12T01:43:28.052031+0000","last_clean":"2023-12-12T01:43:28.052031+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T01:43:28.052031+0000","last_undegraded":"2023-12-12T01:43:28.052031+0000","last_fullsized":"2023-12-12T01:43:28.052031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:43:27.046913+0000","last_change":"2023-12-12T01:43:23.016126+0000","last_active":"2023-12-12T01:43:27.046913+0000","last_peered":"2023-12-12T01:43:27.046913+0000","last_clean":"2023-12-12T01:43:27.046913+0000","last_became_active":"2023-12-12T01:43:23.015425+0000","last_became_peered":"2023-12-12T01:43:23.015425+0000","last_unstale":"2023-12-12T01:43:27.046913+0000","last_undegraded":"2023-12-12T01:43:27.046913+0000","last_fullsized":"2023-12-12T01:43:27.046913+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:20:26.074112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738375,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27140,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344700,"statfs":{"total":96636764160,"available":96608972800,"internally_reserved":0,"allocated":122880,"data_stored":33732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27912,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343928,"statfs":{"total":96636764160,"available":96608182272,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66300000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":700,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":716800,"data_stored":624100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T01:43:31.506 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T01:43:31.774 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:31.775 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T01:43:31.791 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2023-12-12T01:43:31.568920+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":17,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82608,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032912,"statfs":{"total":289910292480,"available":289825701888,"internally_reserved":0,"allocated":1581056,"data_stored":1292366,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.536902"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.051825+0000","last_change":"2023-12-12T01:43:27.051716+0000","last_active":"2023-12-12T01:43:28.051825+0000","last_peered":"2023-12-12T01:43:28.051825+0000","last_clean":"2023-12-12T01:43:28.051825+0000","last_became_active":"2023-12-12T01:43:27.051337+0000","last_became_peered":"2023-12-12T01:43:27.051337+0000","last_unstale":"2023-12-12T01:43:28.051825+0000","last_undegraded":"2023-12-12T01:43:28.051825+0000","last_fullsized":"2023-12-12T01:43:28.051825+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:49:43.136965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052503+0000","last_change":"2023-12-12T01:43:27.055425+0000","last_active":"2023-12-12T01:43:28.052503+0000","last_peered":"2023-12-12T01:43:28.052503+0000","last_clean":"2023-12-12T01:43:28.052503+0000","last_became_active":"2023-12-12T01:43:27.053612+0000","last_became_peered":"2023-12-12T01:43:27.053612+0000","last_unstale":"2023-12-12T01:43:28.052503+0000","last_undegraded":"2023-12-12T01:43:28.052503+0000","last_fullsized":"2023-12-12T01:43:28.052503+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:12:46.742958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052348+0000","last_change":"2023-12-12T01:43:27.053878+0000","last_active":"2023-12-12T01:43:28.052348+0000","last_peered":"2023-12-12T01:43:28.052348+0000","last_clean":"2023-12-12T01:43:28.052348+0000","last_became_active":"2023-12-12T01:43:27.053608+0000","last_became_peered":"2023-12-12T01:43:27.053608+0000","last_unstale":"2023-12-12T01:43:28.052348+0000","last_undegraded":"2023-12-12T01:43:28.052348+0000","last_fullsized":"2023-12-12T01:43:28.052348+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T13:09:28.305200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052188+0000","last_change":"2023-12-12T01:43:27.053430+0000","last_active":"2023-12-12T01:43:28.052188+0000","last_peered":"2023-12-12T01:43:28.052188+0000","last_clean":"2023-12-12T01:43:28.052188+0000","last_became_active":"2023-12-12T01:43:27.053063+0000","last_became_peered":"2023-12-12T01:43:27.053063+0000","last_unstale":"2023-12-12T01:43:28.052188+0000","last_undegraded":"2023-12-12T01:43:28.052188+0000","last_fullsized":"2023-12-12T01:43:28.052188+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:19:33.765966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.319952+0000","last_change":"2023-12-12T01:43:27.050662+0000","last_active":"2023-12-12T01:43:28.319952+0000","last_peered":"2023-12-12T01:43:28.319952+0000","last_clean":"2023-12-12T01:43:28.319952+0000","last_became_active":"2023-12-12T01:43:27.049924+0000","last_became_peered":"2023-12-12T01:43:27.049924+0000","last_unstale":"2023-12-12T01:43:28.319952+0000","last_undegraded":"2023-12-12T01:43:28.319952+0000","last_fullsized":"2023-12-12T01:43:28.319952+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:55:03.243465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.608978+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T01:43:28.608978+0000","last_peered":"2023-12-12T01:43:28.608978+0000","last_clean":"2023-12-12T01:43:28.608978+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T01:43:28.608978+0000","last_undegraded":"2023-12-12T01:43:28.608978+0000","last_fullsized":"2023-12-12T01:43:28.608978+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.609051+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T01:43:28.609051+0000","last_peered":"2023-12-12T01:43:28.609051+0000","last_clean":"2023-12-12T01:43:28.609051+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T01:43:28.609051+0000","last_undegraded":"2023-12-12T01:43:28.609051+0000","last_fullsized":"2023-12-12T01:43:28.609051+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-12-12T01:43:28.052031+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T01:43:28.052031+0000","last_peered":"2023-12-12T01:43:28.052031+0000","last_clean":"2023-12-12T01:43:28.052031+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T01:43:28.052031+0000","last_undegraded":"2023-12-12T01:43:28.052031+0000","last_fullsized":"2023-12-12T01:43:28.052031+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":140,"reported_epoch":14,"state":"active+clean","last_fresh":"2023-12-12T01:43:27.046913+0000","last_change":"2023-12-12T01:43:23.016126+0000","last_active":"2023-12-12T01:43:27.046913+0000","last_peered":"2023-12-12T01:43:27.046913+0000","last_clean":"2023-12-12T01:43:27.046913+0000","last_became_active":"2023-12-12T01:43:23.015425+0000","last_became_peered":"2023-12-12T01:43:23.015425+0000","last_unstale":"2023-12-12T01:43:27.046913+0000","last_undegraded":"2023-12-12T01:43:27.046913+0000","last_fullsized":"2023-12-12T01:43:27.046913+0000","mapping_epoch":10,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:20:26.074112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738376,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27152,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344688,"statfs":{"total":96636764160,"available":96608960512,"internally_reserved":0,"allocated":131072,"data_stored":37210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":724992,"data_stored":627578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T01:43:31.792 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2023-12-12T01:43:31.792 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2023-12-12T01:43:31.792 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2023-12-12T01:43:31.792 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2023-12-12T01:43:32.093 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T01:43:32.093 INFO:teuthology.orchestra.run.smithi140.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-12-12T01:43:32.110 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2023-12-12T01:43:32.111 INFO:teuthology.run_tasks:Running task exec... 2023-12-12T01:43:32.122 INFO:teuthology.task.exec:Executing custom commands... 2023-12-12T01:43:32.122 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T01:43:32.123 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:32.461 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror] 2023-12-12T01:43:32.461 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQB3undlPy07DRAAmYOdFkgOJOHsj2QkR48ftg== 2023-12-12T01:43:32.461 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-12T01:43:32.461 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:32.462 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror 2023-12-12T01:43:32.479 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:32.848 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.0] 2023-12-12T01:43:32.848 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQB3undlyxfDFBAAIMgdzCS6rjxixlP3rEKKtw== 2023-12-12T01:43:32.848 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:32.848 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:32.849 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.0 2023-12-12T01:43:32.867 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:33.205 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.1] 2023-12-12T01:43:33.206 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQB3undl+JPcGRAAv3Cb03jeqTuWi6ig6eTlyg== 2023-12-12T01:43:33.206 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:33.206 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:33.206 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.1 2023-12-12T01:43:33.223 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:33.585 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.2] 2023-12-12T01:43:33.585 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQB3undlITzfHhAABUbJTfVxjeIpGT2dP31sEA== 2023-12-12T01:43:33.585 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:33.586 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:33.586 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.2 2023-12-12T01:43:33.604 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:33.958 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.3] 2023-12-12T01:43:33.958 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQB3undlrH/VIxAAKQUidHishFwb8kV8FSsW0g== 2023-12-12T01:43:33.958 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:33.958 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:33.959 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.3 2023-12-12T01:43:33.976 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:34.335 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror] 2023-12-12T01:43:34.335 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundl1utuMhAAKULIn9Yls8SWlyRCyN4/lQ== 2023-12-12T01:43:34.335 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror-peer" 2023-12-12T01:43:34.336 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:34.336 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror 2023-12-12T01:43:34.355 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:34.693 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.0] 2023-12-12T01:43:34.693 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCfundlI6qqNxAA9JCdyzMWfl98hTjTMRNZzw== 2023-12-12T01:43:34.693 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:34.693 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:34.693 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.0 2023-12-12T01:43:34.713 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:35.051 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.1] 2023-12-12T01:43:35.052 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlqHoBARAAlpAZGEnX+xEle30Kx8nj6g== 2023-12-12T01:43:35.052 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:35.052 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:35.052 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.1 2023-12-12T01:43:35.071 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:35.411 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.2] 2023-12-12T01:43:35.411 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlb9ILBhAAGNVGMFEJ8lgw8HB/uWqvTw== 2023-12-12T01:43:35.412 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:35.412 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:35.412 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.2 2023-12-12T01:43:35.431 DEBUG:teuthology.orchestra.run.smithi140:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2023-12-12T01:43:35.772 INFO:teuthology.orchestra.run.smithi140.stdout:[client.mirror.3] 2023-12-12T01:43:35.772 INFO:teuthology.orchestra.run.smithi140.stdout: key = AQCgundlsVflChAAno4G89BpmZxNREpRpEfNGg== 2023-12-12T01:43:35.772 INFO:teuthology.orchestra.run.smithi140.stdout: caps mon = "profile rbd-mirror" 2023-12-12T01:43:35.772 INFO:teuthology.orchestra.run.smithi140.stdout: caps osd = "profile rbd" 2023-12-12T01:43:35.772 INFO:teuthology.orchestra.run.smithi140.stderr:updated caps for client.mirror.3 2023-12-12T01:43:35.791 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.803 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:43:35.803 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:43:35.805 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:43:35.806 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.817 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T01:43:35.817 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T01:43:35.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T01:43:35.819 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.827 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-12T01:43:35.827 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-12T01:43:35.829 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-12T01:43:35.829 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.838 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-12T01:43:35.839 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-12T01:43:35.841 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-12T01:43:35.841 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T01:43:35.851 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T01:43:35.853 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T01:43:35.853 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T01:43:35.862 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T01:43:35.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T01:43:35.866 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.875 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-12T01:43:35.875 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-12T01:43:35.878 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-12T01:43:35.878 INFO:teuthology.run_tasks:Running task rbd-mirror... 2023-12-12T01:43:35.888 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-12T01:43:35.888 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-12T01:43:35.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-12T01:43:35.891 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-12T01:43:35.903 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702345415 2023-12-12T01:43:35.903 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-12T01:43:35.903 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2023-12-12T01:43:35.914 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1702345415 2023-12-12T01:43:35.914 DEBUG:tasks.rbd_mirror_thrash:Yielding 2023-12-12T01:43:35.914 INFO:teuthology.run_tasks:Running task workunit... 2023-12-12T01:43:35.929 INFO:tasks.workunit:Pulling workunits from ref e7f1b79d320b95939278401b2a5bd1b491f81fe3 2023-12-12T01:43:35.930 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-12-12T01:43:35.930 DEBUG:teuthology.orchestra.run.smithi140:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-12T01:43:35.930 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2023-12-12T01:43:35.930 INFO:tasks.rbd_mirror_thrash:waiting for 70 secs before thrashing 2023-12-12T01:43:35.931 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2023-12-12T01:43:35.931 INFO:tasks.rbd_mirror_thrash:waiting for 112 secs before thrashing 2023-12-12T01:43:35.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T01:43:35.935 INFO:teuthology.orchestra.run.smithi140.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2023-12-12T01:43:35.935 DEBUG:teuthology.orchestra.run.smithi140:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-12T01:43:35.982 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-12T01:43:35.982 DEBUG:teuthology.orchestra.run.smithi140:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2023-12-12T01:43:36.030 INFO:tasks.workunit:timeout=6h 2023-12-12T01:43:36.030 INFO:tasks.workunit:cleanup=True 2023-12-12T01:43:36.031 DEBUG:teuthology.orchestra.run.smithi140:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout e7f1b79d320b95939278401b2a5bd1b491f81fe3 2023-12-12T01:43:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2023-12-12T01:44:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Note: switching to 'e7f1b79d320b95939278401b2a5bd1b491f81fe3'. 2023-12-12T01:44:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:changes and commit them, and you can discard any commits you make in this 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:state without impacting any branches by switching back to a branch. 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:If you want to create a new branch to retain commits you create, you may 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:do so (now or later) by using -c with the switch command. Example: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: git switch -c 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Or undo this operation with: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: git switch - 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-12-12T01:44:19.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:44:19.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:HEAD is now at e7f1b79d320 Merge branch 'rbd-external-migrate' of https://github.com/orozery/ceph into HEAD 2023-12-12T01:44:19.467 DEBUG:teuthology.orchestra.run.smithi140:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2023-12-12T01:44:19.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-12-12T01:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-12T01:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-12-12T01:44:20.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-12-12T01:44:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-12-12T01:44:20.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2023-12-12T01:44:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-12T01:44:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-12-12T01:44:20.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2023-12-12T01:44:20.474 DEBUG:teuthology.orchestra.run.smithi140:> set -ex 2023-12-12T01:44:20.475 DEBUG:teuthology.orchestra.run.smithi140:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2023-12-12T01:44:20.522 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2023-12-12T01:44:20.523 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2023-12-12T01:44:20.523 DEBUG:teuthology.orchestra.run.smithi140:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e7f1b79d320b95939278401b2a5bd1b491f81fe3 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-12T01:44:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ IMAGE_COUNT=50 2023-12-12T01:44:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ export LOCKDEP=0 2023-12-12T01:44:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ type xmlstarlet 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ XMLSTARLET=xmlstarlet 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ RBD_MIRROR_INSTANCES=4 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CLUSTER1=cluster1 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CLUSTER2=cluster2 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ PEER_CLUSTER_SUFFIX= 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ POOL=mirror 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ PARENT_POOL=mirror_parent 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ NS1=ns1 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ NS2=ns2 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ TEMPDIR= 2023-12-12T01:44:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ID=mirror 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ RBD_IMAGE_FEATURES=layering 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ MIRROR_USER_ID_PREFIX=mirror. 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ RBD_MIRROR_MODE=snapshot 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ MIRROR_POOL_MODE=image 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ MIRROR_POOL_MODE=image 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ export CEPH_ARGS=--id mirror 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ LAST_MIRROR_INSTANCE=3 2023-12-12T01:44:20.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2023-12-12T01:44:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2023-12-12T01:44:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2023-12-12T01:44:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_BIN=. 2023-12-12T01:44:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_SRC=. 2023-12-12T01:44:20.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -e CMakeCache.txt ] 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -gt 0 ] 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ setup 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local c 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ trap cleanup $? INT TERM EXIT 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ setup_tempdir 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:20.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mktemp -d 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ TEMPDIR=/tmp/tmp.6PQoIp3L40 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z 1 ] 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ setup_pools cluster1 cluster2 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster2 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mon_map_file 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mon_addr 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local admin_key_file 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid 2023-12-12T01:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror 64 64 2023-12-12T01:44:21.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror' already exists 2023-12-12T01:44:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool create mirror_parent 64 64 2023-12-12T01:44:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror_parent' already exists 2023-12-12T01:44:22.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror 2023-12-12T01:44:25.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 pool init mirror_parent 2023-12-12T01:44:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2023-12-12T01:44:28.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2023-12-12T01:44:28.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2023-12-12T01:44:28.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2023-12-12T01:44:28.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2023-12-12T01:44:28.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2023-12-12T01:44:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z ] 2023-12-12T01:44:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z ] 2023-12-12T01:44:28.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_add cluster1 mirror cluster2 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local client_cluster=cluster2 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster2 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid_var_name 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local error_code 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local peer_uuid 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:44:28.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_uuid=b5fae8d2-fdef-473a-9879-d5a4de07f508 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 17 ] 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 -eq 0 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_add cluster1 mirror_parent cluster2 2023-12-12T01:44:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror_parent 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local client_cluster=cluster2 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster2 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid_var_name 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local error_code 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local peer_uuid 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:44:28.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_uuid=659ab5ea-e42f-41e7-85c7-60d08cca4570 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 17 ] 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 -eq 0 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:28.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ setup_pools cluster2 cluster1 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster1 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mon_map_file 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mon_addr 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local admin_key_file 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid 2023-12-12T01:44:28.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror 64 64 2023-12-12T01:44:29.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror' already exists 2023-12-12T01:44:29.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool create mirror_parent 64 64 2023-12-12T01:44:30.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror_parent' already exists 2023-12-12T01:44:30.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror 2023-12-12T01:44:33.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster2 pool init mirror_parent 2023-12-12T01:44:36.001 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:44:35.996+0000 7f13611da880 -1 rbd::mirror::PoolReplayer: 0x55995d434c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:44:36.000+0000 7f68438af880 -1 rbd::mirror::PoolReplayer: 0x5648cacd0c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.003 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T01:44:36.000+0000 7fa33d240880 -1 rbd::mirror::PoolReplayer: 0x55c8c062cc00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.006 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T01:44:36.004+0000 7f2dfe364880 -1 rbd::mirror::PoolReplayer: 0x55d9f71b4c00 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:44:36.072+0000 7f13611da880 -1 rbd::mirror::PoolReplayer: 0x55995e0fb500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2023-12-12T01:44:36.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T01:44:36.264+0000 7f2dfe364880 -1 rbd::mirror::PoolReplayer: 0x55d9f7ec9500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.271 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:44:36.268+0000 7f68438af880 -1 rbd::mirror::PoolReplayer: 0x5648cad4d500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.279 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T01:44:36.276+0000 7fa33d240880 -1 rbd::mirror::PoolReplayer: 0x55c8c1325500 init: failed to initialize remote pool poller: (2) No such file or directory 2023-12-12T01:44:36.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2023-12-12T01:44:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2023-12-12T01:44:36.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2023-12-12T01:44:36.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2023-12-12T01:44:36.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z ] 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z ] 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_add cluster2 mirror cluster1 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local client_cluster=cluster1 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster1 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid_var_name 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local error_code 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local peer_uuid 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:44:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_uuid=32c5c932-c74a-4ab2-9edd-76da8a021920 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 17 ] 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 -eq 0 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_add cluster2 mirror_parent cluster1 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror_parent 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local client_cluster=cluster1 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local remote_cluster=cluster1 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local uuid_var_name 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local error_code 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local peer_uuid 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:44:36.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2023-12-12T01:44:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ peer_uuid=abf5c14c-4a6c-43bb-96ff-a4cc1166df15 2023-12-12T01:44:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T01:44:36.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 17 ] 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 -eq 0 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirrors cluster1 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:36.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster1:0 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:36.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster1:1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:36.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster1:2 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:44:36.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster1:3 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirrors cluster2 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:36.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster2:0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster2:0 cluster instance 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster2:0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster2:0 ] 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster2 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster2:1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster2:1 cluster instance 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster2:1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster2:1 ] 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster2 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:36.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster2:2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster2:2 cluster instance 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster2:2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster2:2 ] 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:36.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster2:3 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:3 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster2:3 cluster instance 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster2:3 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster2:3 ] 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster2 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster2 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:36.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: add image and test replay after client crashes 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 01:44:36 TEST: add image and test replay after client crashes 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 01:44:36 TEST: add image and test replay after client crashes 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=test 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror test 512M 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:36.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/test 2023-12-12T01:44:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:44:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror test snapshot 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:44:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2023-12-12T01:44:37.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:44:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/test 2023-12-12T01:44:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c261ad10b2 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c261ad10b2 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:36 2023 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:44:36 2023 2023-12-12T01:44:37.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:36 2023 2023-12-12T01:44:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:44:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:44:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:44:37.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:44:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:44:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:44:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:44:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:44:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local instance=cluster1 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:44:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T01:44:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:44:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:38.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:44:38.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:38.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:38.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:44:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:38.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:44:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:38.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:38.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:38.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:44:38.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:38.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:38.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:44:38.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:38.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:44:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:44:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:38.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:44:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:38.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:38.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:38.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:38.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:38.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:38.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:44:38.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:38.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:44:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:38.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:44:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:44:39.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:39.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:39.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:44:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:39.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:44:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:40.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:40.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:44:40.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:40.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:44:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:44:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:40.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:40.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:44:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:40.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:44:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:40.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:44:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:44:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:40.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:40.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:40.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:44:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:40.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:44:40.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:40.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:40.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T01:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:44:42.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:42.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:42.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:42.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:42.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:44:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:42.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:44:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:42.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:42.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:44:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:42.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:42.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:44:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:42.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:44:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:44:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:42.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:42.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:44:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:42.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:44:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:42.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:42.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:44:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:42.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:44:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:42.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T01:44:45.932 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T01:44:45.932 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T01:44:45.933 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-12T01:44:45.933 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:44:45.931+0000 7f135e195640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 40073) UID: 1000 2023-12-12T01:44:46.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:44:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:44:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:46.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:46.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:46.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:44:47.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:47.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:47.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:44:47.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-12T01:44:47.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:47.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:44:47.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:44:47.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:44:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:47.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:44:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:44:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:47.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:44:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:44:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:44:47.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:44:47.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:44:47.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:44:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:47.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:44:47.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:44:47.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:44:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:44:47.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:44:47.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:44:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T01:44:47.933 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T01:44:47.933 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:44:47.934 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T01:44:47.934 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:44:47.934 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:44:47.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:44:47.937 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2023-12-12T01:44:55.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:44:55.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:44:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:44:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:44:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:44:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:55.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:55.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:55.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:44:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:44:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:44:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:44:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:44:55.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:44:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:55.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:44:55.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:44:55.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:44:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:44:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:44:55.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:44:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:44:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:44:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:44:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name= 2023-12-12T01:44:55.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 10 2023-12-12T01:44:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:44:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:44:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:44:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:44:55.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:44:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T01:44:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:44:55.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:44:56.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 618496 bytes 2023-12-12T01:44:56.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 64599040 bytes 2023-12-12T01:44:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 81063936 bytes 2023-12-12T01:44:56.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 139094016 bytes 2023-12-12T01:44:56.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 176670720 bytes 2023-12-12T01:44:56.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 199267328 bytes 2023-12-12T01:44:57.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 322419712 bytes 2023-12-12T01:44:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 335053824 bytes 2023-12-12T01:44:57.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 368690176 bytes 2023-12-12T01:44:57.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 394025984 bytes 2023-12-12T01:44:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 447398912 bytes 2023-12-12T01:44:57.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 472383488 bytes 2023-12-12T01:44:57.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 520580096 bytes 2023-12-12T01:44:58.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 566002688 bytes 2023-12-12T01:44:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 566624256 bytes 2023-12-12T01:44:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 588675072 bytes 2023-12-12T01:44:58.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 595937280 bytes 2023-12-12T01:44:58.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 654107648 bytes 2023-12-12T01:44:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 674506752 bytes 2023-12-12T01:44:58.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 741500928 bytes 2023-12-12T01:44:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 841295872 bytes 2023-12-12T01:44:59.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 894192640 bytes 2023-12-12T01:44:59.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 894818304 bytes 2023-12-12T01:44:59.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 915165184 bytes 2023-12-12T01:44:59.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 965378048 bytes 2023-12-12T01:44:59.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 965991424 bytes 2023-12-12T01:44:59.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1040454656 bytes 2023-12-12T01:45:00.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1148758016 bytes 2023-12-12T01:45:00.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1166022656 bytes 2023-12-12T01:45:00.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1186650112 bytes 2023-12-12T01:45:00.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1236401152 bytes 2023-12-12T01:45:00.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1266915328 bytes 2023-12-12T01:45:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1306450944 bytes 2023-12-12T01:45:01.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1336328192 bytes 2023-12-12T01:45:01.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1392766976 bytes 2023-12-12T01:45:01.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1466138624 bytes 2023-12-12T01:45:01.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1515212800 bytes 2023-12-12T01:45:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1563377664 bytes 2023-12-12T01:45:01.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1616316416 bytes 2023-12-12T01:45:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1642504192 bytes 2023-12-12T01:45:02.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1717174272 bytes 2023-12-12T01:45:02.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1747365888 bytes 2023-12-12T01:45:02.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1786434560 bytes 2023-12-12T01:45:02.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1860371456 bytes 2023-12-12T01:45:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1872870400 bytes 2023-12-12T01:45:02.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1900210176 bytes 2023-12-12T01:45:03.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1955494912 bytes 2023-12-12T01:45:03.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1990451200 bytes 2023-12-12T01:45:03.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2038915072 bytes 2023-12-12T01:45:03.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2046958592 bytes 2023-12-12T01:45:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2161853440 bytes 2023-12-12T01:45:03.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2171471872 bytes 2023-12-12T01:45:04.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2202766336 bytes 2023-12-12T01:45:04.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2267427840 bytes 2023-12-12T01:45:04.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2308935680 bytes 2023-12-12T01:45:04.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2334400512 bytes 2023-12-12T01:45:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2336027648 bytes 2023-12-12T01:45:04.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2384929792 bytes 2023-12-12T01:45:04.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2430534656 bytes 2023-12-12T01:45:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2450973696 bytes 2023-12-12T01:45:05.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2512027648 bytes 2023-12-12T01:45:05.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2567343104 bytes 2023-12-12T01:45:05.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2641447936 bytes 2023-12-12T01:45:05.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2697626624 bytes 2023-12-12T01:45:05.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2709511168 bytes 2023-12-12T01:45:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2722734080 bytes 2023-12-12T01:45:06.282 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T01:45:06.282+0000 7f2dfe364880 -1 rbd::mirror::Mirror: 0x55d9f6563d40 update_pool_replayers: restarting failed pool replayer for uuid: b5fae8d2-fdef-473a-9879-d5a4de07f508 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.286 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:45:06.286+0000 7f68438af880 -1 rbd::mirror::Mirror: 0x5648ca079d40 update_pool_replayers: restarting failed pool replayer for uuid: b5fae8d2-fdef-473a-9879-d5a4de07f508 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.302 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T01:45:06.302+0000 7fa33d240880 -1 rbd::mirror::Mirror: 0x55c8bf9b5e60 update_pool_replayers: restarting failed pool replayer for uuid: b5fae8d2-fdef-473a-9879-d5a4de07f508 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2759775232 bytes 2023-12-12T01:45:06.455 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T01:45:06.454+0000 7f2dfe364880 -1 rbd::mirror::Mirror: 0x55d9f6563d40 update_pool_replayers: restarting failed pool replayer for uuid: 659ab5ea-e42f-41e7-85c7-60d08cca4570 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.466 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:45:06.462+0000 7f68438af880 -1 rbd::mirror::Mirror: 0x5648ca079d40 update_pool_replayers: restarting failed pool replayer for uuid: 659ab5ea-e42f-41e7-85c7-60d08cca4570 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T01:45:06.474+0000 7fa33d240880 -1 rbd::mirror::Mirror: 0x55c8bf9b5e60 update_pool_replayers: restarting failed pool replayer for uuid: 659ab5ea-e42f-41e7-85c7-60d08cca4570 cluster: cluster2 client: client.mirror 2023-12-12T01:45:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2777010176 bytes 2023-12-12T01:45:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:45:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:07.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:45:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:45:07.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:45:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:45:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: down+unknown 2023-12-12T01:45:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: status not found 2023-12-12T01:45:07.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12T01:45:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:45:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:45:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: down+unknown 2023-12-12T01:45:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: status not found 2023-12-12T01:45:07.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12T01:45:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 1 2023-12-12T01:45:07.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:09.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:45:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:45:09.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:45:09.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: down+unknown 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: status not found 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:45:09.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:45:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:45:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:45:09.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:08 2023-12-12T01:45:09.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 1 2023-12-12T01:45:09.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T01:45:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:13.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:13.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:45:13.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:13.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: down+unknown 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: status not found 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:45:13.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:08 2023-12-12T01:45:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 1 2023-12-12T01:45:13.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T01:45:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:45:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:45:21.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:45:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:21.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345477,"remote_snapshot_timestamp":1702345477,"replay_state":"idle"} 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.0 on smithi140 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:18 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:45:21.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:08 2023-12-12T01:45:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:45:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345477,"remote_snapshot_timestamp":1702345477,"replay_state":"idle"} 2023-12-12T01:45:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:45:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep ^up+ 2023-12-12T01:45:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo up+replaying 2023-12-12T01:45:22.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.0 on smithi140 2023-12-12T01:45:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:45:18 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:45:18 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:45:18 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:45:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:45:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:45:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:45:22.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:45:22.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:45:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:45:18 2023-12-12T01:45:22.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:45:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:45:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:45:22.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:45:22.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap1 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap1 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1 2023-12-12T01:45:22.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:45:27.932 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-12T01:45:27.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-12T01:45:27.933 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-12T01:45:27.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T01:45:27.934+0000 7fec4f791640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 40082) UID: 1000 2023-12-12T01:45:32.933 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-12T01:45:32.933 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:45:32.934 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-12T01:45:32.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T01:45:32.934 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T01:45:32.937 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T01:45:32.937 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-12T01:45:37.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:45:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:45:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:45:38.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:45:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:45:38.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T01:45:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:45:38.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:45:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:45:38.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:45:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:45:38.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:45:38.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:45:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:45:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:45:38.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:45:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:45:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:45:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:45:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:45:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:45:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:38.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:45:39.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 6 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:45:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:45:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:45:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=6 2023-12-12T01:45:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:45:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:45:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:45:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:45:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=4 2023-12-12T01:45:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 6 = 4 2023-12-12T01:45:39.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:45:39.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:39.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:45:39.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:45:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:45:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=4 2023-12-12T01:45:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 6 = 4 2023-12-12T01:45:40.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:40.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:45:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:45:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:45:40.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:45:40.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:45:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:45:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:45:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:45:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 6 = 2023-12-12T01:45:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:45:42.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:45:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:45:42.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=6 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 6 = 6 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1 2023-12-12T01:45:42.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:45:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:45:43.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:45:43.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:43.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:45:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:45:43.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap1 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:45:43.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:45:44.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:45:44.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:45:45.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:45:45.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:45:45.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap1-clean 2023-12-12T01:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap1-clean 2023-12-12T01:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:45.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1-clean 2023-12-12T01:45:45.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:45:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:45:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:46.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:45:46.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T01:45:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:45:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:45:46.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 68145152 bytes 2023-12-12T01:45:46.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 138867712 bytes 2023-12-12T01:45:46.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 156357632 bytes 2023-12-12T01:45:47.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 197269504 bytes 2023-12-12T01:45:47.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 252336128 bytes 2023-12-12T01:45:47.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 258461696 bytes 2023-12-12T01:45:47.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 337261568 bytes 2023-12-12T01:45:47.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 412192768 bytes 2023-12-12T01:45:47.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 412783616 bytes 2023-12-12T01:45:48.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 477273088 bytes 2023-12-12T01:45:48.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 533354496 bytes 2023-12-12T01:45:48.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 604564480 bytes 2023-12-12T01:45:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 660324352 bytes 2023-12-12T01:45:48.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 660943872 bytes 2023-12-12T01:45:48.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 718889984 bytes 2023-12-12T01:45:49.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 753059840 bytes 2023-12-12T01:45:49.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 760392704 bytes 2023-12-12T01:45:49.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 760987648 bytes 2023-12-12T01:45:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 786402304 bytes 2023-12-12T01:45:49.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 853637120 bytes 2023-12-12T01:45:49.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 896021504 bytes 2023-12-12T01:45:49.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 912442368 bytes 2023-12-12T01:45:50.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 992235520 bytes 2023-12-12T01:45:50.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1021731840 bytes 2023-12-12T01:45:50.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1074149376 bytes 2023-12-12T01:45:50.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1106552832 bytes 2023-12-12T01:45:50.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1126424576 bytes 2023-12-12T01:45:50.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1160886272 bytes 2023-12-12T01:45:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1196818432 bytes 2023-12-12T01:45:51.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1252308992 bytes 2023-12-12T01:45:51.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1292142592 bytes 2023-12-12T01:45:51.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1319796736 bytes 2023-12-12T01:45:51.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1364525056 bytes 2023-12-12T01:45:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1399548928 bytes 2023-12-12T01:45:51.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1404516352 bytes 2023-12-12T01:45:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1435972608 bytes 2023-12-12T01:45:52.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1479876608 bytes 2023-12-12T01:45:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1505104896 bytes 2023-12-12T01:45:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1524476928 bytes 2023-12-12T01:45:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1560368128 bytes 2023-12-12T01:45:52.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1592148992 bytes 2023-12-12T01:45:52.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1592745984 bytes 2023-12-12T01:45:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1637937152 bytes 2023-12-12T01:45:53.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1700990976 bytes 2023-12-12T01:45:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1747007488 bytes 2023-12-12T01:45:53.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1777049600 bytes 2023-12-12T01:45:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1780905984 bytes 2023-12-12T01:45:53.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1808466944 bytes 2023-12-12T01:45:53.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1855813632 bytes 2023-12-12T01:45:53.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.605811438616085) > weight (0.25) 2023-12-12T01:45:53.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5794423190647982) > weight (0.25) 2023-12-12T01:45:53.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.6130251817702341) > weight (0.25) 2023-12-12T01:45:53.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9320501898254531) > weight (0.25) 2023-12-12T01:45:53.938 INFO:tasks.rbd_mirror_thrash:waiting for 68 secs before thrashing 2023-12-12T01:45:53.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1953342464 bytes 2023-12-12T01:45:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1973368832 bytes 2023-12-12T01:45:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2013144064 bytes 2023-12-12T01:45:54.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2077178880 bytes 2023-12-12T01:45:54.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2081910784 bytes 2023-12-12T01:45:54.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2094319616 bytes 2023-12-12T01:45:54.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2170159104 bytes 2023-12-12T01:45:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2212456448 bytes 2023-12-12T01:45:55.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2248457216 bytes 2023-12-12T01:45:55.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2278753280 bytes 2023-12-12T01:45:55.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2318728192 bytes 2023-12-12T01:45:55.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2338140160 bytes 2023-12-12T01:45:55.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2358946816 bytes 2023-12-12T01:45:55.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2406782976 bytes 2023-12-12T01:45:55.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2447142912 bytes 2023-12-12T01:45:56.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2486763520 bytes 2023-12-12T01:45:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2532970496 bytes 2023-12-12T01:45:56.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2579533824 bytes 2023-12-12T01:45:56.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2594944000 bytes 2023-12-12T01:45:56.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2658560000 bytes 2023-12-12T01:45:56.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2679211008 bytes 2023-12-12T01:45:56.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2688796672 bytes 2023-12-12T01:45:57.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 2743487488 bytes 2023-12-12T01:45:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 2764468224 bytes 2023-12-12T01:45:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:45:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:45:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:45:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:57.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:57.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:58.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:45:58.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":17871496.53,"bytes_per_snapshot":268072448.0,"last_snapshot_bytes":536144896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345539,"remote_snapshot_timestamp":1702345539,"replay_state":"idle"} 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.0 on smithi140 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:48 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:45:58.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:45:36 2023-12-12T01:45:58.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:45:58.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":17871496.53,"bytes_per_snapshot":268072448.0,"last_snapshot_bytes":536144896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345539,"remote_snapshot_timestamp":1702345539,"replay_state":"idle"} 2023-12-12T01:45:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:45:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:45:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:45:58.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:45:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.0 on smithi140 2023-12-12T01:45:58.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:45:48 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:45:48 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:45:48 2023-12-12T01:45:58.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:45:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:45:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:45:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:45:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:45:58.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:45:58.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:45:58.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:45:58.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:45:48 2023-12-12T01:45:58.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:45:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:45:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:45:58.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:45:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:45:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:45:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap2 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap2 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2 2023-12-12T01:45:58.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:46:27.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:46:27.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:46:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:46:28.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:46:28.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:46:28.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:46:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:46:28.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:46:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:46:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:46:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:46:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:46:28.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:46:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:46:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:46:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:46:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:46:28.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:46:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo "state": "Replaying" 2023-12-12T01:46:28.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T01:46:28.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:46:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:46:28.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + sed -e s|[/ ]|_|g 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo cluster2-mirror-test.status 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:46:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:28.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:46:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 9 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:46:30.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:46:30.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:46:30.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:46:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=9 2023-12-12T01:46:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:46:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:46:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:46:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:46:30.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 9 = 2023-12-12T01:46:30.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:46:30.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:46:30.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:46:30.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:46:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:46:30.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:46:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:46:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:46:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 9 = 2023-12-12T01:46:30.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:46:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:31.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:46:31.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:46:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:46:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 9 = 2023-12-12T01:46:31.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:33.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:46:33.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:46:33.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:46:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=9 2023-12-12T01:46:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 9 = 9 2023-12-12T01:46:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:33.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:46:33.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:46:34.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2023-12-12T01:46:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:46:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:46:34.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 4 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:46:34.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap1-clean ] 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1-clean 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:46:34.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:46:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:34.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:46:35.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:46:35.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:36.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:46:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap2 2023-12-12T01:46:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:37.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2 2023-12-12T01:46:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:46:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:46:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:37.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:46:37.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:46:37.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:46:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap2-clean 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap2-clean 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2-clean 2023-12-12T01:46:39.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:46:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:46:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:40.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:46:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T01:46:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:46:40.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:46:40.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 22810624 bytes 2023-12-12T01:46:40.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 77356032 bytes 2023-12-12T01:46:40.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 132469760 bytes 2023-12-12T01:46:41.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 175735808 bytes 2023-12-12T01:46:41.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 242465792 bytes 2023-12-12T01:46:41.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 260956160 bytes 2023-12-12T01:46:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 340614144 bytes 2023-12-12T01:46:41.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 392870912 bytes 2023-12-12T01:46:41.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 445196288 bytes 2023-12-12T01:46:41.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 479151104 bytes 2023-12-12T01:46:42.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 515687424 bytes 2023-12-12T01:46:42.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 555099136 bytes 2023-12-12T01:46:42.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 621448192 bytes 2023-12-12T01:46:42.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 703337472 bytes 2023-12-12T01:46:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 762832896 bytes 2023-12-12T01:46:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 869488640 bytes 2023-12-12T01:46:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 891241472 bytes 2023-12-12T01:46:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 967739392 bytes 2023-12-12T01:46:43.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1014782976 bytes 2023-12-12T01:46:43.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1056479232 bytes 2023-12-12T01:46:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1076943872 bytes 2023-12-12T01:46:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1095769088 bytes 2023-12-12T01:46:44.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1100897280 bytes 2023-12-12T01:46:44.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1134752768 bytes 2023-12-12T01:46:44.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1179955200 bytes 2023-12-12T01:46:44.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1240160256 bytes 2023-12-12T01:46:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1254886400 bytes 2023-12-12T01:46:44.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1328112640 bytes 2023-12-12T01:46:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1385968640 bytes 2023-12-12T01:46:45.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:46:45.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:46:46.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:46:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:46:46.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:46:46.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":17871496.53,"bytes_per_snapshot":268072448.0,"last_snapshot_bytes":536144896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345539,"remote_snapshot_timestamp":1702345539,"replay_state":"idle"} 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.0 on smithi140 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:46:18 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:46:46.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:46:36 2023-12-12T01:46:46.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:46:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":17871496.53,"bytes_per_snapshot":268072448.0,"last_snapshot_bytes":536144896,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345539,"remote_snapshot_timestamp":1702345539,"replay_state":"idle"} 2023-12-12T01:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:46:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:46:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.0 on smithi140 2023-12-12T01:46:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:46:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:46:18 2023-12-12T01:46:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:46:18 2023-12-12T01:46:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:46:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:46.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:46:18 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:46:46.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:46:46.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:46:46.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:46:46.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:46:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:46:18 2023-12-12T01:46:46.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:46:46.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:46:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:46:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap3 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap3 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:46:46.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:46:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:46:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3 2023-12-12T01:46:46.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:47:01.939 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T01:47:01.940 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T01:47:01.940 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T01:47:01.941 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:47:01.940+0000 7f0e8871b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 41199) UID: 1000 2023-12-12T01:47:08.941 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T01:47:08.941 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:47:08.941 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T01:47:08.942 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:47:08.942 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:47:08.945 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:47:08.945 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2023-12-12T01:47:14.938 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6818408982492241) > weight (0.25) 2023-12-12T01:47:14.938 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T01:47:14.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T01:47:14.939 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T01:47:14.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T01:47:14.936+0000 7f2ea50cf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 40085) UID: 1000 2023-12-12T01:47:15.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:47:15.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:47:16.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:47:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T01:47:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T01:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:16.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:16.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:16.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:16.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:16.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:16.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:16.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:16.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:16.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:16.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:16.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:16.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:16.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:16.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:16.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:16.604 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:47:16.602+0000 7f97b9f54640 -1 reset not still connected to 0x560d14f4be10 2023-12-12T01:47:16.604 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T01:47:16.606+0000 7f97b9f54640 -1 reset not still connected to 0x560d151c28f0 2023-12-12T01:47:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:16.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:16.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:47:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:17.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:17.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:17.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:17.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:17.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:17.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:17.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:17.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:17.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:17.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:18.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:18.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:18.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:18.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:18.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T01:47:18.939 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T01:47:18.940 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:47:18.940 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T01:47:18.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T01:47:18.940 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T01:47:18.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T01:47:18.943 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-12-12T01:47:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:20.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:20.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:20.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:20.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:20.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:20.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:20.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:20.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:20.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:20.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:20.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:20.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:20.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:20.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:20.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:20.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:20.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:20.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:20.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T01:47:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:24.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:24.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:24.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:24.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:24.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:24.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:24.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:24.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:24.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:24.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:24.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:24.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:24.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:25.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:25.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:25.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:25.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:25.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:25.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:25.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:25.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T01:47:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:33.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:33.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:33.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:33.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:33.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:33.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:33.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:33.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:33.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:33.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:33.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:33.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:33.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:33.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:33.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:33.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:33.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:33.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:33.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:33.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:33.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T01:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:47:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:47:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:41.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:47:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:47:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:47:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:47:41.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:47:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:41.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:41.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:47:41.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:41.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:47:41.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:47:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:47:41.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:47:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:42.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:47:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:47:42.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:47:42.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:47:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:42.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:47:42.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:47:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:47:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:47:42.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:42.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:47:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:47:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:47:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:47:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:47:42.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:47:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:47:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:47:42.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:42.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:47:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T01:47:42.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T01:47:42.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:47:42.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:42.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:42.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:47:42.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:42.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 13 2023-12-12T01:47:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:47:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:44.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:47:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:47:44.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:47:44.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:47:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=13 2023-12-12T01:47:44.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:47:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:47:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:47:44.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:47:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:47:44.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:47:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:47:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:47:44.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:47:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:47:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 13 = 2023-12-12T01:47:44.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:47:44.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:47:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:47:45.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 13 = 2023-12-12T01:47:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:47:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:47:45.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:47:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:47:45.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 13 = 2023-12-12T01:47:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:47:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:47:47.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=13 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 13 = 13 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:47:47.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:47:48.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2023-12-12T01:47:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:47:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:47:48.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 6 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:46:35 2023 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:47:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap2-clean ] 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2-clean 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:47:49.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:47:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:50.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:47:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:50.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap3 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:51.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:51.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:47:51.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:47:51.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:52.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:47:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:53.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:47:53.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap3-clean 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap3-clean 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3-clean 2023-12-12T01:47:53.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:47:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:47:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:47:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:47:54.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:47:54.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:47:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=12 2023-12-12T01:47:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:47:54.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:47:54.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 19110912 bytes 2023-12-12T01:47:54.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 67088384 bytes 2023-12-12T01:47:54.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 129785856 bytes 2023-12-12T01:47:54.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 167078912 bytes 2023-12-12T01:47:55.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 208195584 bytes 2023-12-12T01:47:55.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 292288512 bytes 2023-12-12T01:47:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 350117888 bytes 2023-12-12T01:47:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 412968960 bytes 2023-12-12T01:47:55.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 426761216 bytes 2023-12-12T01:47:56.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 468425728 bytes 2023-12-12T01:47:56.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 506690560 bytes 2023-12-12T01:47:56.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 557136896 bytes 2023-12-12T01:47:56.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 631921664 bytes 2023-12-12T01:47:56.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 646641664 bytes 2023-12-12T01:47:56.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 709224448 bytes 2023-12-12T01:47:56.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 737736704 bytes 2023-12-12T01:47:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 824648704 bytes 2023-12-12T01:47:57.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 881816576 bytes 2023-12-12T01:47:57.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 893161472 bytes 2023-12-12T01:47:57.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 923191296 bytes 2023-12-12T01:47:57.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 948792320 bytes 2023-12-12T01:47:57.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 981501952 bytes 2023-12-12T01:47:58.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 982132736 bytes 2023-12-12T01:47:58.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1019515904 bytes 2023-12-12T01:47:58.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1040731136 bytes 2023-12-12T01:47:58.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1102013440 bytes 2023-12-12T01:47:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1163331584 bytes 2023-12-12T01:47:58.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1188243456 bytes 2023-12-12T01:47:58.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1207960576 bytes 2023-12-12T01:47:58.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1214835712 bytes 2023-12-12T01:47:59.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1250673664 bytes 2023-12-12T01:47:59.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1325776896 bytes 2023-12-12T01:47:59.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1349826560 bytes 2023-12-12T01:47:59.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1503908864 bytes 2023-12-12T01:47:59.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1515384832 bytes 2023-12-12T01:48:00.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1524047872 bytes 2023-12-12T01:48:00.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1579202560 bytes 2023-12-12T01:48:00.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1605531648 bytes 2023-12-12T01:48:00.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1683722240 bytes 2023-12-12T01:48:00.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1722342400 bytes 2023-12-12T01:48:00.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1752117248 bytes 2023-12-12T01:48:01.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1828938752 bytes 2023-12-12T01:48:01.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1829569536 bytes 2023-12-12T01:48:01.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1888579584 bytes 2023-12-12T01:48:01.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1894962176 bytes 2023-12-12T01:48:01.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1915871232 bytes 2023-12-12T01:48:01.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1995698176 bytes 2023-12-12T01:48:01.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1996309504 bytes 2023-12-12T01:48:01.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2001186816 bytes 2023-12-12T01:48:02.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2028187648 bytes 2023-12-12T01:48:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2029198336 bytes 2023-12-12T01:48:02.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2064177152 bytes 2023-12-12T01:48:02.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2086104064 bytes 2023-12-12T01:48:02.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2111468544 bytes 2023-12-12T01:48:02.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2136180736 bytes 2023-12-12T01:48:02.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2156371968 bytes 2023-12-12T01:48:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2156985344 bytes 2023-12-12T01:48:03.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2222082048 bytes 2023-12-12T01:48:03.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2233723904 bytes 2023-12-12T01:48:03.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2266318848 bytes 2023-12-12T01:48:03.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2343758848 bytes 2023-12-12T01:48:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2401514496 bytes 2023-12-12T01:48:03.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2462336000 bytes 2023-12-12T01:48:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2555455488 bytes 2023-12-12T01:48:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2602238976 bytes 2023-12-12T01:48:04.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2626067456 bytes 2023-12-12T01:48:04.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2691336192 bytes 2023-12-12T01:48:04.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2739714048 bytes 2023-12-12T01:48:04.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2788900864 bytes 2023-12-12T01:48:05.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2838718464 bytes 2023-12-12T01:48:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2901313536 bytes 2023-12-12T01:48:05.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 2937653248 bytes 2023-12-12T01:48:05.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 2988120064 bytes 2023-12-12T01:48:05.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3004410880 bytes 2023-12-12T01:48:06.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 3067962368 bytes 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:48:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:48:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:48:07.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:48:07.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:48:07.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":16726425.6,"bytes_per_snapshot":501792768.0,"last_snapshot_bytes":501792768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345663,"remote_snapshot_timestamp":1702345663,"replay_state":"idle"} 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:48:06 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:48:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:48:06 2023-12-12T01:48:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:48:07.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":16726425.6,"bytes_per_snapshot":501792768.0,"last_snapshot_bytes":501792768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345663,"remote_snapshot_timestamp":1702345663,"replay_state":"idle"} 2023-12-12T01:48:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:48:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:48:07.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:48:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:48:07.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:48:07.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:48:06 2023-12-12T01:48:07.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:48:06 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:48:06 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:48:07.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:48:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:48:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:48:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:48:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:48:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:48:06 2023-12-12T01:48:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:48:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:48:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:48:07.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:48:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:48:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:07.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap4 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap4 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4 2023-12-12T01:48:07.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:48:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:48:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:48:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:48:37.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grepshift -i 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:48:37.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:48:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:48:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:48:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:48:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:48:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:48:37.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:48:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:48:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:48:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:48:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:48:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:48:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:48:37.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:48:37.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:48:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:48:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:48:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:48:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:48:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:48:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:48:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:48:37.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:48:37.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:48:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:48:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:48:37.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:48:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:48:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:48:37.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:48:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:48:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:48:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:48:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:48:37.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:48:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:48:37.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:48:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:48:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:48:37.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:48:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:48:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:48:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:48:37.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:48:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:48:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:48:37.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:48:37.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:48:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:48:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:48:37.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:48:37.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:48:37.944 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.39349823201395484) > weight (0.25) 2023-12-12T01:48:37.944 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T01:48:37.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T01:48:37.945 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-12T01:48:37.946 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T01:48:37.947+0000 7efdf897c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 42923) UID: 1000 2023-12-12T01:48:39.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 17 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:48:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:48:39.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:48:39.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:48:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=17 2023-12-12T01:48:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:48:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:48:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:48:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:48:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:48:39.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:48:39.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 17 = 2023-12-12T01:48:39.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:48:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:48:40.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 17 = 2023-12-12T01:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:48:40.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:48:40.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:48:40.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 17 = 2023-12-12T01:48:40.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:48:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:42.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:48:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:48:42.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:48:42.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=17 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 17 = 17 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4 2023-12-12T01:48:42.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:48:42.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:48:43.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2023-12-12T01:48:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:48:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:48:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:48:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 8 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:47:49 2023 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:48:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap3-clean ] 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3-clean 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:48:43.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:48:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:43.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:48:43.946 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T01:48:43.946 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:48:43.946 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T01:48:43.947 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T01:48:43.947 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T01:48:43.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T01:48:43.949 INFO:tasks.rbd_mirror_thrash:waiting for 117 secs before thrashing 2023-12-12T01:48:44.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:48:44.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:45.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:48:45.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:45.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap4 2023-12-12T01:48:45.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:45.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:48:46.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:48:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:47.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:48:47.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:48.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap4-clean 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap4-clean 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4-clean 2023-12-12T01:48:48.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:48:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:48:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:48:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T01:48:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:48:49.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:48:49.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 49265664 bytes 2023-12-12T01:48:49.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 105724928 bytes 2023-12-12T01:48:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 175694848 bytes 2023-12-12T01:48:50.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 198837248 bytes 2023-12-12T01:48:50.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 218203136 bytes 2023-12-12T01:48:50.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 278424576 bytes 2023-12-12T01:48:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 359974912 bytes 2023-12-12T01:48:50.904 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 461298688 bytes 2023-12-12T01:48:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 532396032 bytes 2023-12-12T01:48:51.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 573791232 bytes 2023-12-12T01:48:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 623572992 bytes 2023-12-12T01:48:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 676737024 bytes 2023-12-12T01:48:51.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 768752640 bytes 2023-12-12T01:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 812319744 bytes 2023-12-12T01:48:52.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 910977024 bytes 2023-12-12T01:48:52.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 923667456 bytes 2023-12-12T01:48:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 951027712 bytes 2023-12-12T01:48:52.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 1015043072 bytes 2023-12-12T01:48:52.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1086308352 bytes 2023-12-12T01:48:52.946 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T01:48:52.946 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T01:48:52.946 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T01:48:52.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:48:52.943+0000 7f33ca462640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42791) UID: 1000 2023-12-12T01:48:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1093727232 bytes 2023-12-12T01:48:53.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1147795456 bytes 2023-12-12T01:48:53.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1205052416 bytes 2023-12-12T01:48:53.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1233246208 bytes 2023-12-12T01:48:53.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1291140096 bytes 2023-12-12T01:48:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1330384896 bytes 2023-12-12T01:48:54.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1384156160 bytes 2023-12-12T01:48:54.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1440494592 bytes 2023-12-12T01:48:54.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1445476352 bytes 2023-12-12T01:48:54.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1503289344 bytes 2023-12-12T01:48:54.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1567678464 bytes 2023-12-12T01:48:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1639329792 bytes 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:48:55.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:48:55.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:48:56.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:48:56.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:48:56.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:48:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:48:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:48:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":501792768.0,"last_snapshot_bytes":501792768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345663,"remote_snapshot_timestamp":1702345663,"replay_state":"idle"} 2023-12-12T01:48:56.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:48:36 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:48:56.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:48:36 2023-12-12T01:48:56.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:48:56.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":501792768.0,"last_snapshot_bytes":501792768,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345663,"remote_snapshot_timestamp":1702345663,"replay_state":"idle"} 2023-12-12T01:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:48:56.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:48:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:48:36 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:48:36 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:48:36 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:48:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:48:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:48:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:48:56.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:48:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:48:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:48:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:48:36 2023-12-12T01:48:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:48:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:48:56.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:48:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:48:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap5 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap5 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5 2023-12-12T01:48:56.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:48:59.947 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T01:48:59.947 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:48:59.947 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T01:48:59.947 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:48:59.947 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:48:59.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:48:59.950 INFO:tasks.rbd_mirror_thrash:waiting for 104 secs before thrashing 2023-12-12T01:49:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:49:25.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:49:25.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T01:49:26.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:49:26.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:49:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:49:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:49:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:49:26.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:49:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:49:26.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:49:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:49:26.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:49:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:49:26.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:49:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:49:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:49:26.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:49:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:49:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:49:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:49:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:49:26.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:49:26.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:49:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:49:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:49:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:49:26.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:49:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:49:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:49:26.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:49:26.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:49:26.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:49:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:49:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:49:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:49:26.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:49:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:49:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:49:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:49:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:49:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:49:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:49:26.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:49:26.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:49:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:49:26.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:49:26.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:49:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:49:26.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:49:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:49:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:49:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:49:26.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:49:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:49:26.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:26.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:49:28.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 21 2023-12-12T01:49:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:49:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:49:28.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:49:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=21 2023-12-12T01:49:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:49:28.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:49:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:49:28.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:49:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 21 = 2023-12-12T01:49:28.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:49:28.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:49:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:49:29.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:49:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:49:29.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:49:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:49:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:49:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 21 = 2023-12-12T01:49:29.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:29.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:49:29.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:49:29.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:49:29.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:49:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:49:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:49:29.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:49:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:49:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 21 = 2023-12-12T01:49:29.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:49:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:49:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:49:31.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:49:31.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=21 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 21 = 21 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5 2023-12-12T01:49:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:49:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:49:32.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2023-12-12T01:49:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:49:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:49:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:49:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 10 2023-12-12T01:49:32.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:47:49 2023 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:49:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap4-clean ] 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4-clean 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:49:33.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:49:33.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:34.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:49:34.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:34.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:34.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap5 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:34.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:49:35.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:49:35.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:36.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:49:36.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:36.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:49:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap5-clean 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap5-clean 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5-clean 2023-12-12T01:49:37.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:49:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:49:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:37.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:49:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T01:49:37.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:49:37.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:49:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 31208448 bytes 2023-12-12T01:49:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 89808896 bytes 2023-12-12T01:49:37.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 125545472 bytes 2023-12-12T01:49:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 142266368 bytes 2023-12-12T01:49:38.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 155710464 bytes 2023-12-12T01:49:38.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 166984704 bytes 2023-12-12T01:49:38.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 189813760 bytes 2023-12-12T01:49:38.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 227684352 bytes 2023-12-12T01:49:38.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 304147456 bytes 2023-12-12T01:49:38.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 342960128 bytes 2023-12-12T01:49:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 343543808 bytes 2023-12-12T01:49:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 393212928 bytes 2023-12-12T01:49:39.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 408686592 bytes 2023-12-12T01:49:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 489025536 bytes 2023-12-12T01:49:39.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 581188608 bytes 2023-12-12T01:49:39.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 610403328 bytes 2023-12-12T01:49:40.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 684359680 bytes 2023-12-12T01:49:40.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 710699008 bytes 2023-12-12T01:49:40.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 711310336 bytes 2023-12-12T01:49:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 756685824 bytes 2023-12-12T01:49:40.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 807578624 bytes 2023-12-12T01:49:40.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 848609280 bytes 2023-12-12T01:49:40.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 878996480 bytes 2023-12-12T01:49:41.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 937220096 bytes 2023-12-12T01:49:41.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 953475072 bytes 2023-12-12T01:49:41.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1000366080 bytes 2023-12-12T01:49:41.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1006408704 bytes 2023-12-12T01:49:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1036063744 bytes 2023-12-12T01:49:41.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1075988480 bytes 2023-12-12T01:49:41.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1110782976 bytes 2023-12-12T01:49:42.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1139338240 bytes 2023-12-12T01:49:42.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1159600128 bytes 2023-12-12T01:49:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1160929280 bytes 2023-12-12T01:49:42.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1257564160 bytes 2023-12-12T01:49:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1293806592 bytes 2023-12-12T01:49:42.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1302617088 bytes 2023-12-12T01:49:42.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1334418432 bytes 2023-12-12T01:49:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1364486144 bytes 2023-12-12T01:49:43.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1382215680 bytes 2023-12-12T01:49:43.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1449594880 bytes 2023-12-12T01:49:43.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1467595776 bytes 2023-12-12T01:49:43.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1481076736 bytes 2023-12-12T01:49:43.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1493982208 bytes 2023-12-12T01:49:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1517606912 bytes 2023-12-12T01:49:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1550540800 bytes 2023-12-12T01:49:43.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1558334464 bytes 2023-12-12T01:49:44.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1584887808 bytes 2023-12-12T01:49:44.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1652901888 bytes 2023-12-12T01:49:44.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1719813120 bytes 2023-12-12T01:49:44.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1746244608 bytes 2023-12-12T01:49:44.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1833979904 bytes 2023-12-12T01:49:45.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 1854498816 bytes 2023-12-12T01:49:45.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 1945375744 bytes 2023-12-12T01:49:45.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 1972547584 bytes 2023-12-12T01:49:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2015419392 bytes 2023-12-12T01:49:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2042700800 bytes 2023-12-12T01:49:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2066799616 bytes 2023-12-12T01:49:45.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2096833536 bytes 2023-12-12T01:49:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2137863168 bytes 2023-12-12T01:49:46.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2170134528 bytes 2023-12-12T01:49:46.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2186068992 bytes 2023-12-12T01:49:46.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2215550976 bytes 2023-12-12T01:49:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2228194304 bytes 2023-12-12T01:49:46.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2289740800 bytes 2023-12-12T01:49:46.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2352404480 bytes 2023-12-12T01:49:47.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2405131264 bytes 2023-12-12T01:49:47.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2454463488 bytes 2023-12-12T01:49:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2472714240 bytes 2023-12-12T01:49:47.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2495767552 bytes 2023-12-12T01:49:47.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2522274816 bytes 2023-12-12T01:49:47.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2552359936 bytes 2023-12-12T01:49:48.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 2633610240 bytes 2023-12-12T01:49:48.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 2716685312 bytes 2023-12-12T01:49:48.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 2754337792 bytes 2023-12-12T01:49:48.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 2794112000 bytes 2023-12-12T01:49:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7600 IOs, 2801540096 bytes 2023-12-12T01:49:48.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7700 IOs, 2824148992 bytes 2023-12-12T01:49:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7800 IOs, 2879933440 bytes 2023-12-12T01:49:49.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7900 IOs, 2929236992 bytes 2023-12-12T01:49:49.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8000 IOs, 3000111104 bytes 2023-12-12T01:49:49.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8100 IOs, 3084561408 bytes 2023-12-12T01:49:49.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8200 IOs, 3142216704 bytes 2023-12-12T01:49:49.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8300 IOs, 3163004928 bytes 2023-12-12T01:49:50.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8400 IOs, 3184002048 bytes 2023-12-12T01:49:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:49:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:49:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:49:50.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:49:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:49:51.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:49:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:49:51.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":17212450.13,"bytes_per_snapshot":522302464.0,"last_snapshot_bytes":516373504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345767,"remote_snapshot_timestamp":1702345767,"replay_state":"idle"} 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:49:51.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:49:36 2023-12-12T01:49:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:49:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:49:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:49:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:49:51.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:49:36 2023-12-12T01:49:51.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:49:51.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":17212450.13,"bytes_per_snapshot":522302464.0,"last_snapshot_bytes":516373504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345767,"remote_snapshot_timestamp":1702345767,"replay_state":"idle"} 2023-12-12T01:49:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:49:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:49:51.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:49:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:49:51.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:49:51.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:49:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:49:36 2023-12-12T01:49:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:49:36 2023-12-12T01:49:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:49:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:51.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:49:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:49:36 2023-12-12T01:49:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:49:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:49:51.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:49:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:49:51.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:49:51.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:49:51.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:49:51.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:49:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:49:36 2023-12-12T01:49:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:49:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:49:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:49:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap6 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap6 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:49:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6 2023-12-12T01:49:51.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:50:20.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:50:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:50:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:50:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:50:21.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:50:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:50:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:50:21.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:50:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:50:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:50:21.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:50:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:50:21.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:50:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:50:21.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:50:21.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:50:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:50:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:50:21.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:50:21.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:50:21.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:50:21.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:50:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:50:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:50:21.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:50:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:50:21.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:50:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:50:21.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:50:21.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:50:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:50:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:50:21.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:50:21.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:50:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:50:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:50:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:50:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:50:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:50:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:50:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:50:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:50:21.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:50:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:50:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:50:21.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:50:21.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:50:21.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:50:21.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:50:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:50:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:50:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:50:21.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:21.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:50:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 25 2023-12-12T01:50:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:50:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:23.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:50:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:50:23.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:50:23.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:50:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=25 2023-12-12T01:50:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:50:23.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:50:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:50:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:50:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:50:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 25 = 2023-12-12T01:50:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:50:23.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:50:23.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:50:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:50:24.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:50:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:50:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:50:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:50:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:50:24.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 25 = 2023-12-12T01:50:24.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:50:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:50:24.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:50:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:50:24.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:50:24.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:50:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:50:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:50:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:50:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 25 = 2023-12-12T01:50:24.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:50:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:50:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:50:26.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:50:26.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=25 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 25 = 25 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:50:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:50:27.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 12 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:50:27.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:49:33 2023 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:50:27.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:50:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap5-clean ] 2023-12-12T01:50:27.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5-clean 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:27.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:50:28.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:50:28.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:29.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:50:29.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:29.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap6 2023-12-12T01:50:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:30.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6 2023-12-12T01:50:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:50:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:50:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:30.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:50:30.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:50:30.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:31.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:50:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:32.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:50:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap6-clean 2023-12-12T01:50:32.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap6-clean 2023-12-12T01:50:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6-clean 2023-12-12T01:50:32.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:50:33.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:50:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:33.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:50:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T01:50:33.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:50:33.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:50:33.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 46505984 bytes 2023-12-12T01:50:33.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 96380928 bytes 2023-12-12T01:50:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 122646528 bytes 2023-12-12T01:50:34.056 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 184893440 bytes 2023-12-12T01:50:34.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 238808064 bytes 2023-12-12T01:50:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 340395008 bytes 2023-12-12T01:50:34.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 375141376 bytes 2023-12-12T01:50:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 399222784 bytes 2023-12-12T01:50:35.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 453262336 bytes 2023-12-12T01:50:35.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 519258112 bytes 2023-12-12T01:50:35.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 547947520 bytes 2023-12-12T01:50:35.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 559190016 bytes 2023-12-12T01:50:35.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 559794176 bytes 2023-12-12T01:50:35.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 627651584 bytes 2023-12-12T01:50:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 630024192 bytes 2023-12-12T01:50:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 661730304 bytes 2023-12-12T01:50:36.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 666760192 bytes 2023-12-12T01:50:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 704957440 bytes 2023-12-12T01:50:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 757753856 bytes 2023-12-12T01:50:36.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 797596672 bytes 2023-12-12T01:50:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 812722176 bytes 2023-12-12T01:50:36.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 817083392 bytes 2023-12-12T01:50:36.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 851582976 bytes 2023-12-12T01:50:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 939931648 bytes 2023-12-12T01:50:37.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 972264448 bytes 2023-12-12T01:50:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1013197824 bytes 2023-12-12T01:50:37.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1061950464 bytes 2023-12-12T01:50:37.771 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1106310144 bytes 2023-12-12T01:50:37.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1123469312 bytes 2023-12-12T01:50:38.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1177147392 bytes 2023-12-12T01:50:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1213519872 bytes 2023-12-12T01:50:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1297018880 bytes 2023-12-12T01:50:38.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1316329472 bytes 2023-12-12T01:50:38.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1341594624 bytes 2023-12-12T01:50:38.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1389863936 bytes 2023-12-12T01:50:39.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1437597696 bytes 2023-12-12T01:50:39.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1438188544 bytes 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:39.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:50:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:50:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:50:39.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:50:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:50:40.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":17860471.47,"bytes_per_snapshot":526093824.0,"last_snapshot_bytes":535814144,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702345822,"remote_snapshot_timestamp":1702345822,"replay_state":"idle"} 2023-12-12T01:50:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:50:36 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:50:40.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:50:36 2023-12-12T01:50:40.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:50:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":17860471.47,"bytes_per_snapshot":526093824.0,"last_snapshot_bytes":535814144,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1702345822,"remote_snapshot_timestamp":1702345822,"replay_state":"idle"} 2023-12-12T01:50:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:50:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo up+replaying 2023-12-12T01:50:40.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep ^up+ 2023-12-12T01:50:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:50:40.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:50:40.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:50:36 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:50:36 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:50:36 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:50:40.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:50:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:50:40.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:50:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:50:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:50:40.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:50:40.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:50:36 2023-12-12T01:50:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:50:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:50:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:50:40.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:50:40.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap7 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap7 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7 2023-12-12T01:50:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:50:40.950 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.930928169244251) > weight (0.25) 2023-12-12T01:50:40.950 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5168018234528123) > weight (0.25) 2023-12-12T01:50:40.950 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-12T01:50:40.951 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-12T01:50:40.951 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-12T01:50:40.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T01:50:40.954+0000 7f6dc7b6b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 40089) UID: 1000 2023-12-12T01:50:43.952 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7199529172795784) > weight (0.25) 2023-12-12T01:50:43.952 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T01:50:43.952 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T01:50:43.952 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-12T01:50:43.953 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:50:43.954+0000 7f684086a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 40075) UID: 1000 2023-12-12T01:50:48.951 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-12T01:50:48.951 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:50:48.952 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-12T01:50:48.952 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-12T01:50:48.952 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-12T01:50:48.955 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-12T01:50:48.955 INFO:tasks.rbd_mirror_thrash:waiting for 119 secs before thrashing 2023-12-12T01:50:51.953 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T01:50:51.954 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:50:51.954 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T01:50:51.954 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T01:50:51.954 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T01:50:51.957 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T01:50:51.957 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-12T01:51:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:51:09.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:51:10.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemonsgrep cluster1 -i rbd state 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: mirror status mirror/test 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:51:10.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:51:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:51:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:51:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:51:10.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:51:10.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:51:10.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:51:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:51:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:51:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:51:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:51:10.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:51:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:51:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:51:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:51:10.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:51:10.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:51:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:51:10.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:51:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:51:10.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:51:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:51:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:51:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:51:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:51:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:51:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:51:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:51:10.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:51:10.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:51:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:51:10.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:51:10.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:51:10.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:51:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:51:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:51:10.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:51:10.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:51:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:51:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:51:10.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:51:10.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:51:10.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:51:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:51:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:51:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:51:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:51:10.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:51:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:51:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:51:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:51:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:51:10.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:51:10.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:51:10.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 29 2023-12-12T01:51:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:51:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:12.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:51:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:51:12.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:51:12.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:51:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=29 2023-12-12T01:51:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:51:12.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:51:12.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:51:12.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:51:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:51:12.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:51:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:51:12.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:51:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:51:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:51:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 29 = 2023-12-12T01:51:12.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:51:13.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:51:13.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:51:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:51:13.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:51:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:51:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:51:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:51:13.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:51:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 29 = 2023-12-12T01:51:13.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:51:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:51:14.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:51:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:51:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:51:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:51:14.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:51:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 29 = 2023-12-12T01:51:14.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:51:15.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:51:15.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:51:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=29 2023-12-12T01:51:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 29 = 29 2023-12-12T01:51:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:51:15.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:51:16.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 14 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:51:16.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:49:33 2023 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:51:16.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap6-clean ] 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6-clean 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:16.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:16.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:51:17.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:51:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:18.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:51:18.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:18.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap7 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:19.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:51:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:51:19.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:21.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:51:21.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:21.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap7-clean 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap7-clean 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:21.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7-clean 2023-12-12T01:51:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:51:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:51:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:22.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:51:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T01:51:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:51:22.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:51:22.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 43491328 bytes 2023-12-12T01:51:22.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 144572416 bytes 2023-12-12T01:51:22.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 181399552 bytes 2023-12-12T01:51:23.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 217057280 bytes 2023-12-12T01:51:23.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 217671680 bytes 2023-12-12T01:51:23.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 268251136 bytes 2023-12-12T01:51:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 308848640 bytes 2023-12-12T01:51:23.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 359846912 bytes 2023-12-12T01:51:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 430702592 bytes 2023-12-12T01:51:24.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 456788992 bytes 2023-12-12T01:51:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 481035264 bytes 2023-12-12T01:51:24.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 500403200 bytes 2023-12-12T01:51:24.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 520921088 bytes 2023-12-12T01:51:24.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 529212416 bytes 2023-12-12T01:51:24.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 536770560 bytes 2023-12-12T01:51:24.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 564792320 bytes 2023-12-12T01:51:25.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 620863488 bytes 2023-12-12T01:51:25.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 678458368 bytes 2023-12-12T01:51:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 680975360 bytes 2023-12-12T01:51:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 717551616 bytes 2023-12-12T01:51:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 774600704 bytes 2023-12-12T01:51:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 819745792 bytes 2023-12-12T01:51:26.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 872779776 bytes 2023-12-12T01:51:26.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 914603008 bytes 2023-12-12T01:51:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 959893504 bytes 2023-12-12T01:51:26.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1040226304 bytes 2023-12-12T01:51:26.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1076956160 bytes 2023-12-12T01:51:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1121224704 bytes 2023-12-12T01:51:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1139636224 bytes 2023-12-12T01:51:27.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1203758080 bytes 2023-12-12T01:51:27.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1263872000 bytes 2023-12-12T01:51:27.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1344599040 bytes 2023-12-12T01:51:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1357646848 bytes 2023-12-12T01:51:28.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1404901376 bytes 2023-12-12T01:51:28.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1482051584 bytes 2023-12-12T01:51:28.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1517742080 bytes 2023-12-12T01:51:28.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1566678016 bytes 2023-12-12T01:51:28.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1636966400 bytes 2023-12-12T01:51:28.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1637579776 bytes 2023-12-12T01:51:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1712761856 bytes 2023-12-12T01:51:29.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1772046336 bytes 2023-12-12T01:51:29.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1807604736 bytes 2023-12-12T01:51:29.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1874362368 bytes 2023-12-12T01:51:29.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1914388480 bytes 2023-12-12T01:51:30.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1951291392 bytes 2023-12-12T01:51:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:51:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:51:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:51:30.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:51:30.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:51:31.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:51:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:51:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345822,"remote_snapshot_timestamp":1702345822,"replay_state":"idle"} 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:51:06 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:51:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:51:06 2023-12-12T01:51:31.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:51:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345822,"remote_snapshot_timestamp":1702345822,"replay_state":"idle"} 2023-12-12T01:51:31.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:51:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:51:31.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:51:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:51:31.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:51:31.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:51:06 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:51:06 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:51:06 2023-12-12T01:51:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:51:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:51:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:51:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:51:31.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:51:31.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:51:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:51:31.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:51:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:51:06 2023-12-12T01:51:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:51:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:51:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:51:31.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:51:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap8 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap8 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8 2023-12-12T01:51:31.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:52:00.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:52:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:52:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:52:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:52:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:52:01.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ localgrep instance=cluster1 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i+ statelocal loop_instance 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:52:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:52:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:52:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:01.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:52:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:52:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:52:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:52:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:52:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:01.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:52:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:52:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:52:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:01.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:52:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:01.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:52:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:52:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:52:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T01:52:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T01:52:01.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:52:01.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:52:01.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:52:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:52:01.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:01.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:52:03.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 33 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:03.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:52:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:03.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:52:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=33 2023-12-12T01:52:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:03.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:03.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:03.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:03.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 33 = 2023-12-12T01:52:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:52:04.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:04.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:04.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:04.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:04.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 33 = 2023-12-12T01:52:04.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:52:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:04.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:04.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:05.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 33 = 2023-12-12T01:52:05.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:06.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=33 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 33 = 33 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:06.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8 2023-12-12T01:52:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:52:06.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:52:07.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2023-12-12T01:52:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:52:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:52:07.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 16 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:51:17 2023 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:52:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:52:07.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:52:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap7-clean ] 2023-12-12T01:52:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-12T01:52:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7-clean 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:52:08.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:52:08.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:09.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:52:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:09.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap8 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:10.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:10.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:52:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:52:10.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:11.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:52:11.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:12.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap8-clean 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap8-clean 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8-clean 2023-12-12T01:52:12.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:52:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:52:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:13.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:52:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:52:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:52:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:52:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 59807744 bytes 2023-12-12T01:52:13.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 76797952 bytes 2023-12-12T01:52:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 139235328 bytes 2023-12-12T01:52:14.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 163927040 bytes 2023-12-12T01:52:14.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 187000832 bytes 2023-12-12T01:52:14.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 231146496 bytes 2023-12-12T01:52:14.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 303458304 bytes 2023-12-12T01:52:15.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 349762560 bytes 2023-12-12T01:52:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 357824512 bytes 2023-12-12T01:52:15.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 450440192 bytes 2023-12-12T01:52:15.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 487156736 bytes 2023-12-12T01:52:15.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 533520384 bytes 2023-12-12T01:52:15.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 608119808 bytes 2023-12-12T01:52:16.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 655418368 bytes 2023-12-12T01:52:16.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 664476672 bytes 2023-12-12T01:52:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 668352512 bytes 2023-12-12T01:52:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 695628800 bytes 2023-12-12T01:52:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 711364608 bytes 2023-12-12T01:52:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 789583872 bytes 2023-12-12T01:52:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 791235584 bytes 2023-12-12T01:52:17.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 833816576 bytes 2023-12-12T01:52:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 853130240 bytes 2023-12-12T01:52:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 874125312 bytes 2023-12-12T01:52:17.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 898328576 bytes 2023-12-12T01:52:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 927535104 bytes 2023-12-12T01:52:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1033346048 bytes 2023-12-12T01:52:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1113660416 bytes 2023-12-12T01:52:18.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1226764288 bytes 2023-12-12T01:52:18.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1272123392 bytes 2023-12-12T01:52:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1302486016 bytes 2023-12-12T01:52:18.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1320183808 bytes 2023-12-12T01:52:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1379049472 bytes 2023-12-12T01:52:19.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1452533760 bytes 2023-12-12T01:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1501392896 bytes 2023-12-12T01:52:19.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1526319104 bytes 2023-12-12T01:52:19.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1557756928 bytes 2023-12-12T01:52:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1598089216 bytes 2023-12-12T01:52:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1674057728 bytes 2023-12-12T01:52:20.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1750812672 bytes 2023-12-12T01:52:20.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1781219328 bytes 2023-12-12T01:52:20.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1802776576 bytes 2023-12-12T01:52:20.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1808099328 bytes 2023-12-12T01:52:20.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1852659712 bytes 2023-12-12T01:52:21.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1927254016 bytes 2023-12-12T01:52:21.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1990610944 bytes 2023-12-12T01:52:21.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1991255040 bytes 2023-12-12T01:52:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2041987072 bytes 2023-12-12T01:52:21.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2085174272 bytes 2023-12-12T01:52:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2171762688 bytes 2023-12-12T01:52:22.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2216779776 bytes 2023-12-12T01:52:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2255822848 bytes 2023-12-12T01:52:22.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2306461696 bytes 2023-12-12T01:52:22.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2369176576 bytes 2023-12-12T01:52:22.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2445538304 bytes 2023-12-12T01:52:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2530373632 bytes 2023-12-12T01:52:23.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:52:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:52:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:52:24.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:52:24.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:52:24.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":14698666.67,"bytes_per_snapshot":509137920.0,"last_snapshot_bytes":520126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345871,"remote_snapshot_timestamp":1702345922,"replay_state":"syncing","seconds_until_synced":34,"syncing_percent":100,"syncing_snapshot_timestamp":1702345922} 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:52:06 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+stopped 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: local image is primary 2023-12-12T01:52:24.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:52:06 2023-12-12T01:52:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":14698666.67,"bytes_per_snapshot":509137920.0,"last_snapshot_bytes":520126464,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345871,"remote_snapshot_timestamp":1702345922,"replay_state":"syncing","seconds_until_synced":34,"syncing_percent":100,"syncing_snapshot_timestamp":1702345922} 2023-12-12T01:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:52:24.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:52:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:52:24.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:52:24.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:52:06 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:52:06 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:52:06 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:52:24.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:52:24.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:52:24.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:52:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:52:24.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:52:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:52:06 2023-12-12T01:52:24.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:52:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:52:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:52:24.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:52:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:52:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:24.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap9 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap9 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9 2023-12-12T01:52:24.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:52:30.958 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.28929780958230156) > weight (0.25) 2023-12-12T01:52:30.958 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7518659772612208) > weight (0.25) 2023-12-12T01:52:30.958 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-12T01:52:30.959 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-12T01:52:30.959 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-12T01:52:30.960 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T01:52:30.960+0000 7fa33a1fb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40077) UID: 1000 2023-12-12T01:52:38.960 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-12T01:52:38.960 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:52:38.961 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-12T01:52:38.961 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-12T01:52:38.961 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-12T01:52:38.964 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-12T01:52:38.964 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-12T01:52:47.956 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.30738395389362594) > weight (0.25) 2023-12-12T01:52:47.956 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9812110344876552) > weight (0.25) 2023-12-12T01:52:47.956 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9347243749324469) > weight (0.25) 2023-12-12T01:52:47.956 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2023-12-12T01:52:47.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2023-12-12T01:52:47.957 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T01:52:47.958 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T01:52:47.960+0000 7fbd06713640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 40110) UID: 1000 2023-12-12T01:52:51.961 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2023-12-12T01:52:51.961 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:52:51.962 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2023-12-12T01:52:51.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2023-12-12T01:52:51.962 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2023-12-12T01:52:51.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2023-12-12T01:52:51.965 INFO:tasks.rbd_mirror_thrash:waiting for 92 secs before thrashing 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:52:53.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:52:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:52:53.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:52:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:52:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:52:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + sleep 0 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:52:54.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:52:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:52:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:52:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:52:54.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:52:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:52:54.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:52:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:54.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:52:54.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:52:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:54.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:52:54.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:54.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:52:54.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:52:54.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:52:54.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:52:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:52:54.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:54.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:52:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:52:54.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:52:54.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:52:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:52:54.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:52:54.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:52:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:52:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:52:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:52:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:54.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:52:56.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 37 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:52:56.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:52:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=37 2023-12-12T01:52:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:52:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:56.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 37 = 2023-12-12T01:52:56.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:52:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:57.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:57.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 37 = 2023-12-12T01:52:57.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:52:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:58.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:52:58.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:52:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 37 = 2023-12-12T01:52:58.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:52:59.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:52:59.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:52:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=37 2023-12-12T01:52:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 37 = 37 2023-12-12T01:52:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:52:59.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:52:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:53:00.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2023-12-12T01:53:00.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 18 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:51:17 2023 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:53:00.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:53:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:53:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:53:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:53:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap8-clean ] 2023-12-12T01:53:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-12T01:53:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8-clean 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:00.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:01.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:02.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:02.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap9 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:03.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:03.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:03.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:05.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap9-clean 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap9-clean 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:05.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9-clean 2023-12-12T01:53:05.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:53:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror test 2023-12-12T01:53:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:06.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:53:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T01:53:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:53:06.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2023-12-12T01:53:06.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 31057920 bytes 2023-12-12T01:53:06.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 59506688 bytes 2023-12-12T01:53:06.967 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 140486656 bytes 2023-12-12T01:53:07.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 202945536 bytes 2023-12-12T01:53:07.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 226985984 bytes 2023-12-12T01:53:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 296849408 bytes 2023-12-12T01:53:07.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 383490048 bytes 2023-12-12T01:53:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 433556480 bytes 2023-12-12T01:53:08.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 449349632 bytes 2023-12-12T01:53:08.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 501732352 bytes 2023-12-12T01:53:08.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 502346752 bytes 2023-12-12T01:53:08.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 510599168 bytes 2023-12-12T01:53:08.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 563295232 bytes 2023-12-12T01:53:08.711 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 572801024 bytes 2023-12-12T01:53:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 603430912 bytes 2023-12-12T01:53:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 673754112 bytes 2023-12-12T01:53:09.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 734700544 bytes 2023-12-12T01:53:09.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 797535232 bytes 2023-12-12T01:53:09.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 801548288 bytes 2023-12-12T01:53:09.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 830312448 bytes 2023-12-12T01:53:09.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 832472064 bytes 2023-12-12T01:53:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 862491648 bytes 2023-12-12T01:53:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 892993536 bytes 2023-12-12T01:53:10.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 908247040 bytes 2023-12-12T01:53:10.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 962594816 bytes 2023-12-12T01:53:10.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1018844160 bytes 2023-12-12T01:53:10.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1057781760 bytes 2023-12-12T01:53:10.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1083409408 bytes 2023-12-12T01:53:11.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1188451328 bytes 2023-12-12T01:53:11.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1259507712 bytes 2023-12-12T01:53:11.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1318219776 bytes 2023-12-12T01:53:11.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1368118272 bytes 2023-12-12T01:53:12.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1468983296 bytes 2023-12-12T01:53:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1498650624 bytes 2023-12-12T01:53:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1548585984 bytes 2023-12-12T01:53:12.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1615357952 bytes 2023-12-12T01:53:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1644902400 bytes 2023-12-12T01:53:12.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1730941952 bytes 2023-12-12T01:53:13.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1731538944 bytes 2023-12-12T01:53:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1777839104 bytes 2023-12-12T01:53:13.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1823556608 bytes 2023-12-12T01:53:13.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1824190464 bytes 2023-12-12T01:53:13.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1866692608 bytes 2023-12-12T01:53:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1911638016 bytes 2023-12-12T01:53:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1941614592 bytes 2023-12-12T01:53:14.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1982883840 bytes 2023-12-12T01:53:14.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2013241344 bytes 2023-12-12T01:53:14.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2036321280 bytes 2023-12-12T01:53:14.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2123848704 bytes 2023-12-12T01:53:14.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2156467200 bytes 2023-12-12T01:53:15.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2243859456 bytes 2023-12-12T01:53:15.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2338770944 bytes 2023-12-12T01:53:15.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2353560576 bytes 2023-12-12T01:53:15.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2367411200 bytes 2023-12-12T01:53:15.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2411291648 bytes 2023-12-12T01:53:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2485641216 bytes 2023-12-12T01:53:16.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2502795264 bytes 2023-12-12T01:53:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2521414656 bytes 2023-12-12T01:53:16.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2570802176 bytes 2023-12-12T01:53:16.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2607073280 bytes 2023-12-12T01:53:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2624637952 bytes 2023-12-12T01:53:16.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2661054464 bytes 2023-12-12T01:53:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2691766272 bytes 2023-12-12T01:53:17.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2763741184 bytes 2023-12-12T01:53:17.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2867501056 bytes 2023-12-12T01:53:17.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2890533888 bytes 2023-12-12T01:53:17.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2972346368 bytes 2023-12-12T01:53:17.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 3027294208 bytes 2023-12-12T01:53:18.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 3053196288 bytes 2023-12-12T01:53:18.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 3118390272 bytes 2023-12-12T01:53:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 3143033856 bytes 2023-12-12T01:53:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3164576768 bytes 2023-12-12T01:53:18.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3185587200 bytes 2023-12-12T01:53:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3227573248 bytes 2023-12-12T01:53:19.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 3271494656 bytes 2023-12-12T01:53:19.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7600 IOs, 3335598080 bytes 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:53:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:53:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:53:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:53:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying 2023-12-12T01:53:20.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local description_pattern= 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local service_pattern= 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:53:20.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= rbd --cluster cluster1 mirror image status mirror/test 2023-12-12T01:53:20.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:test: 2023-12-12T01:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: global_id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:53:20.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: up+replaying 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: replaying, {"bytes_per_second":17572966.4,"bytes_per_snapshot":527188992.0,"last_snapshot_bytes":527188992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345975,"remote_snapshot_timestamp":1702345975,"replay_state":"idle"} 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: service: mirror.3 on smithi140 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12 01:53:06 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: peer_sites: 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: name: cluster2 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state: down+unknown 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: description: status not found 2023-12-12T01:53:20.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: last_update: 2023-12-12T01:53:20.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ state: .*up+replaying /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: state: up+replaying 2023-12-12T01:53:20.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^ description: .* /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: description: replaying, {"bytes_per_second":17572966.4,"bytes_per_snapshot":527188992.0,"last_snapshot_bytes":527188992,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1702345975,"remote_snapshot_timestamp":1702345975,"replay_state":"idle"} 2023-12-12T01:53:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:53:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:53:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep ^up+ 2023-12-12T01:53:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:53:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep service: *mirror..* on /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: service: mirror.3 on smithi140 2023-12-12T01:53:20.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -nEe s/^ last_update: *(.*) *$/\1/p /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.mirror_status 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update=2023-12-12 01:53:06 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying 2023-12-12 01:53:06 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state_pattern=up+replaying 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local prev_last_update=2023-12-12 01:53:06 2023-12-12T01:53:20.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror.mirror_status 2023-12-12T01:53:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror.mirror_status 2023-12-12T01:53:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:53:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror.mirror_status 2023-12-12T01:53:20.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2023-12-12T01:53:20.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T01:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_update state 2023-12-12T01:53:20.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/last_update 2023-12-12T01:53:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ last_update=2023-12-12 01:53:06 2023-12-12T01:53:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //images/image[name='test']/state 2023-12-12T01:53:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=up+replaying 2023-12-12T01:53:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo up+replaying 2023-12-12T01:53:20.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep up+replaying 2023-12-12T01:53:20.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:up+replaying 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap10 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap10 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10 2023-12-12T01:53:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:53:47.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:53:47.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:53:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:53:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:53:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:53:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ localgrep instance=cluster1 -i 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:53:48.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:53:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:53:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:53:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:53:48.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:53:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:53:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:53:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:53:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:53:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:53:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:53:48.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:53:48.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:53:48.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:53:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:53:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:53:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:53:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:53:48.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:53:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:53:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:53:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:53:48.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:53:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:53:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:53:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:53:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:53:48.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:53:48.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:53:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:53:48.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:53:48.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:53:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:53:48.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:53:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:53:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:49.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:53:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:53:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:53:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:53:49.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:53:49.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:53:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:53:49.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:49.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:53:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 41 2023-12-12T01:53:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:53:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:53:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:53:50.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:53:50.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:53:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=41 2023-12-12T01:53:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:53:50.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:53:50.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:53:50.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:53:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:53:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:53:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:53:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:53:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:53:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:53:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 41 = 2023-12-12T01:53:51.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:51.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:53:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:53:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:53:51.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:53:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:53:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:53:51.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 41 = 2023-12-12T01:53:51.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:52.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:53:52.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:53:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:53:52.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 41 = 2023-12-12T01:53:52.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:53:54.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:53:54.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:53:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=41 2023-12-12T01:53:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 41 = 41 2023-12-12T01:53:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2023-12-12T01:53:54.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:53:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10 2023-12-12T01:53:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:53:54.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:53:55.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 20 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:53:55.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:53:01 2023 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:53:55.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snap9-clean ] 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9-clean 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:55.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:55.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:55.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:55.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:57.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:57.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:57.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap10 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:57.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:57.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:53:58.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:58.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:59.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:53:59.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:59.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ clean_snap_name=snap10-clean 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror test snap10-clean 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:53:59.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10-clean 2023-12-12T01:53:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror test 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:54:00.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror test started 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T01:54:01.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/test 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local+ cluster_instance=cluster1 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep+ -ilocal state 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: cluster=cluster1 2023-12-12T01:54:01.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T01:54:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T01:54:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:54:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T01:54:01.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/test 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:54:01.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T01:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T01:54:01.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:54:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:54:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T01:54:01.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2023-12-12T01:54:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:54:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:54:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/test 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:54:01.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:54:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T01:54:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:54:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:54:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T01:54:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T01:54:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T01:54:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:54:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:54:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T01:54:01.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2023-12-12T01:54:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:54:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:54:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/test 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T01:54:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T01:54:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T01:54:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T01:54:01.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:54:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:54:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T01:54:01.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2023-12-12T01:54:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/test'] 2023-12-12T01:54:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T01:54:01.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T01:54:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/test 2023-12-12T01:54:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T01:54:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:54:01.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:01.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T01:54:01.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T01:54:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:54:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:54:01.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T01:54:01.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2023-12-12T01:54:01.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T01:54:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T01:54:01.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T01:54:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T01:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T01:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T01:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2023-12-12T01:54:01.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-test.status 2023-12-12T01:54:01.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-test.status 2023-12-12T01:54:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:54:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:54:01.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-test.status 2023-12-12T01:54:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-test.status 2023-12-12T01:54:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T01:54:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:01.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror test 2023-12-12T01:54:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:01.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2023-12-12T01:54:03.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 44 2023-12-12T01:54:03.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.status 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2023-12-12T01:54:03.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:54:03.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T01:54:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=44 2023-12-12T01:54:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:54:03.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T01:54:03.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:54:03.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:54:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:54:03.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 44 = 2023-12-12T01:54:03.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T01:54:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:04.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:54:04.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:54:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T01:54:04.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:54:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T01:54:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T01:54:04.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T01:54:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T01:54:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 44 = 2023-12-12T01:54:04.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.status 2023-12-12T01:54:05.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2023-12-12T01:54:05.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T01:54:05.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=44 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 44 = 44 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10-clean 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:54:05.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T01:54:06.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2023-12-12T01:54:06.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'test': 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 21 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10c647fc4af1 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10c647fc4af1 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:53:01 2023 2023-12-12T01:54:06.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:44:50 2023 2023-12-12T01:54:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T01:54:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:54:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:54:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 57dea94f-38eb-4616-859e-d8fa39e52fc4 2023-12-12T01:54:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T01:54:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 10 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap1 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap1 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:06.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:06.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:07.965 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T01:54:07.965 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T01:54:07.965 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T01:54:07.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:54:07.967+0000 7f6dd0c90640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 44002) UID: 1000 2023-12-12T01:54:08.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:08.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap1-clean 2023-12-12T01:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap1-clean 2023-12-12T01:54:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap1-clean 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:09.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:09.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:10.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:10.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:10.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:10.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap2 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap2 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:11.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:11.966 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T01:54:11.966 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:54:11.967 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T01:54:11.967 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:54:11.967 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:54:11.970 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:54:11.970 INFO:tasks.rbd_mirror_thrash:waiting for 88 secs before thrashing 2023-12-12T01:54:12.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:12.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:12.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap2-clean 2023-12-12T01:54:12.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap2-clean 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap2-clean 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:12.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:13.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:13.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:14.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:14.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:15.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap3 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap3 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:15.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:15.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:16.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:16.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:17.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:17.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap3-clean 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap3-clean 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap3-clean 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:17.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:17.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:18.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:19.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap4 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap4 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:20.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:20.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:20.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:20.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:21.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:21.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:22.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:22.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap4-clean 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap4-clean 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap4-clean 2023-12-12T01:54:22.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:22.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:23.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:23.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:23.966 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6946577465204724) > weight (0.25) 2023-12-12T01:54:23.966 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5977123043133465) > weight (0.25) 2023-12-12T01:54:23.966 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.46379055022956317) > weight (0.25) 2023-12-12T01:54:23.966 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5641065958172012) > weight (0.25) 2023-12-12T01:54:23.966 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2023-12-12T01:54:24.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:24.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap5 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap5 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:24.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5 2023-12-12T01:54:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:24.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:25.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:25.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:26.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:26.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap5-clean 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap5-clean 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap5-clean 2023-12-12T01:54:27.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:27.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:27.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:28.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:29.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap6 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap6 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:29.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:30.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:30.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:31.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:31.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:31.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap6-clean 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap6-clean 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap6-clean 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:32.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:32.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:33.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:33.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap7 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap7 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:33.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:34.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:34.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:35.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:35.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:36.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap7-clean 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap7-clean 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap7-clean 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:36.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:36.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:36.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:38.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:38.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap8 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap8 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:38.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:40.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:40.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap8-clean 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap8-clean 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap8-clean 2023-12-12T01:54:40.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:40.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:40.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:41.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:41.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:42.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap9 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap9 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9 2023-12-12T01:54:43.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:43.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:43.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:43.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:45.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:45.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:45.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap9-clean 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap9-clean 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap9-clean 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:45.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:45.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:46.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:46.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:47.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:47.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:47.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap10 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap10 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:47.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:48.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:48.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:49.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:49.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap10-clean 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror test snap10-clean 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap10-clean 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:49.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export 2023-12-12T01:54:50.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:50.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:51.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2023-12-12T01:54:51.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:52.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-test.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-test.export 2023-12-12T01:54:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:54:52.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 10 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap1 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap1 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap1 2023-12-12T01:54:52.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2023-12-12T01:54:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap1-clean 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap1-clean 2023-12-12T01:54:52.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2023-12-12T01:54:53.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:53.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap2 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap2 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap2 2023-12-12T01:54:53.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2023-12-12T01:54:54.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap2-clean 2023-12-12T01:54:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2023-12-12T01:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap2-clean 2023-12-12T01:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2023-12-12T01:54:55.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap3 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap3 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap3 2023-12-12T01:54:55.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2023-12-12T01:54:56.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap3-clean 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap3-clean 2023-12-12T01:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2023-12-12T01:54:57.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap4 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap4 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap4 2023-12-12T01:54:57.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2023-12-12T01:54:58.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap4-clean 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap4-clean 2023-12-12T01:54:58.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2023-12-12T01:54:59.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap5 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap5 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:54:59.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap5 2023-12-12T01:54:59.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2023-12-12T01:55:00.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap5-clean 2023-12-12T01:55:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2023-12-12T01:55:00.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap5-clean 2023-12-12T01:55:00.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2023-12-12T01:55:01.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap6 2023-12-12T01:55:01.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap6 2023-12-12T01:55:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap6 2023-12-12T01:55:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2023-12-12T01:55:02.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap6-clean 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap6-clean 2023-12-12T01:55:02.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2023-12-12T01:55:03.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap7 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap7 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap7 2023-12-12T01:55:03.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2023-12-12T01:55:04.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap7-clean 2023-12-12T01:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2023-12-12T01:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:04.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap7-clean 2023-12-12T01:55:04.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2023-12-12T01:55:05.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap8 2023-12-12T01:55:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap8 2023-12-12T01:55:05.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap8 2023-12-12T01:55:05.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2023-12-12T01:55:06.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap8-clean 2023-12-12T01:55:06.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2023-12-12T01:55:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap8-clean 2023-12-12T01:55:06.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2023-12-12T01:55:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap9 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap9 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap9 2023-12-12T01:55:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2023-12-12T01:55:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap9-clean 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap9-clean 2023-12-12T01:55:08.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2023-12-12T01:55:09.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap10 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap10 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap10 2023-12-12T01:55:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2023-12-12T01:55:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap10-clean 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap10-clean 2023-12-12T01:55:10.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2023-12-12T01:55:11.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T01:55:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror test 2023-12-12T01:55:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:11.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T01:55:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror test 2023-12-12T01:55:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:11.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2023-12-12T01:55:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/test 2023-12-12T01:55:11.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T01:55:11.630+0000 7f631930d640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562358e8b040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T01:55:11.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T01:55:11.630+0000 7f631fb1a640 -1 rbd::mirror::ImageReplayer: 0x5623582ba280 [3/57dea94f-38eb-4616-859e-d8fa39e52fc4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T01:55:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror test deleted 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T01:55:12.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror test 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-12T01:55:12.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T01:55:12.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-12T01:55:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T01:55:12.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T01:55:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror test deleted 2023-12-12T01:55:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:55:12.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror test 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=test 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/test 2023-12-12T01:55:12.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T01:55:12.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image test: (2) No such file or directory 2023-12-12T01:55:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T01:55:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T01:55:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T01:55:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: create many images 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 01:55:12 TEST: create many images 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 01:55:12 TEST: create many images 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ snap_name=snap 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 50 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_1 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_1 128M 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:12.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_1 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:13.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-12T01:55:13.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:55:13.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_1': 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 126d9007d013 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.126d9007d013 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:13 2023 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:13 2023 2023-12-12T01:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:13 2023 2023-12-12T01:55:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:55:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:55:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: b834df41-6596-47b9-95f6-b6243a57f231 2023-12-12T01:55:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T01:55:13.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:55:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T01:55:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:55:13.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:55:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 67013632 bytes 2023-12-12T01:55:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 122907648 bytes 2023-12-12T01:55:13.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 129786880 bytes 2023-12-12T01:55:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 181625856 bytes 2023-12-12T01:55:14.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 241587200 bytes 2023-12-12T01:55:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 266491904 bytes 2023-12-12T01:55:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 333546496 bytes 2023-12-12T01:55:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 392961024 bytes 2023-12-12T01:55:15.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 440936448 bytes 2023-12-12T01:55:15.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 470145024 bytes 2023-12-12T01:55:15.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 510021632 bytes 2023-12-12T01:55:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 556502016 bytes 2023-12-12T01:55:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 611113984 bytes 2023-12-12T01:55:15.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 622670848 bytes 2023-12-12T01:55:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 680415232 bytes 2023-12-12T01:55:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 737077248 bytes 2023-12-12T01:55:16.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 740082688 bytes 2023-12-12T01:55:16.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 815000576 bytes 2023-12-12T01:55:16.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 844531712 bytes 2023-12-12T01:55:16.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 935746560 bytes 2023-12-12T01:55:17.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 952183808 bytes 2023-12-12T01:55:17.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 972343296 bytes 2023-12-12T01:55:17.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 995259392 bytes 2023-12-12T01:55:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1011931136 bytes 2023-12-12T01:55:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1034574848 bytes 2023-12-12T01:55:17.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1080001536 bytes 2023-12-12T01:55:18.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1179014144 bytes 2023-12-12T01:55:18.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1199515648 bytes 2023-12-12T01:55:18.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1253965824 bytes 2023-12-12T01:55:18.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1316352000 bytes 2023-12-12T01:55:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1353409536 bytes 2023-12-12T01:55:18.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1431079936 bytes 2023-12-12T01:55:19.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1444375552 bytes 2023-12-12T01:55:19.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1471678464 bytes 2023-12-12T01:55:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1510207488 bytes 2023-12-12T01:55:19.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1527135232 bytes 2023-12-12T01:55:19.771 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1623769088 bytes 2023-12-12T01:55:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1641717760 bytes 2023-12-12T01:55:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1664349184 bytes 2023-12-12T01:55:20.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1713976320 bytes 2023-12-12T01:55:20.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1769479168 bytes 2023-12-12T01:55:20.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1785798656 bytes 2023-12-12T01:55:20.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1859702784 bytes 2023-12-12T01:55:20.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1908940800 bytes 2023-12-12T01:55:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1919646720 bytes 2023-12-12T01:55:21.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1966865408 bytes 2023-12-12T01:55:21.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1991344128 bytes 2023-12-12T01:55:21.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2018816000 bytes 2023-12-12T01:55:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2049622016 bytes 2023-12-12T01:55:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2086642688 bytes 2023-12-12T01:55:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2092435456 bytes 2023-12-12T01:55:22.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2127416320 bytes 2023-12-12T01:55:22.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2166857728 bytes 2023-12-12T01:55:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:55:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:55:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:55:22.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_2 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_2 128M 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:22.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_2 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:22.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-12T01:55:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:55:23.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-12T01:55:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_2': 2023-12-12T01:55:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1276f8d2229d 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1276f8d2229d 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:22 2023 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:22 2023 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:22 2023 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:55:23.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d1652be7-048c-4b97-9aef-5e3f0ab7f525 2023-12-12T01:55:23.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:55:23.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-12T01:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T01:55:23.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:55:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T01:55:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:55:23.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:55:23.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 37211136 bytes 2023-12-12T01:55:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 95879168 bytes 2023-12-12T01:55:23.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 200943616 bytes 2023-12-12T01:55:23.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 236722176 bytes 2023-12-12T01:55:24.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 288043008 bytes 2023-12-12T01:55:24.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 297168896 bytes 2023-12-12T01:55:24.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 317636608 bytes 2023-12-12T01:55:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 339945472 bytes 2023-12-12T01:55:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 390136832 bytes 2023-12-12T01:55:24.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 421664768 bytes 2023-12-12T01:55:25.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 442324992 bytes 2023-12-12T01:55:25.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 566290432 bytes 2023-12-12T01:55:25.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 613373952 bytes 2023-12-12T01:55:25.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 701315072 bytes 2023-12-12T01:55:25.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 723682304 bytes 2023-12-12T01:55:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 792780800 bytes 2023-12-12T01:55:26.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 858361856 bytes 2023-12-12T01:55:26.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 895767552 bytes 2023-12-12T01:55:26.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 938982400 bytes 2023-12-12T01:55:26.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 965102592 bytes 2023-12-12T01:55:27.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 977505280 bytes 2023-12-12T01:55:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1013993472 bytes 2023-12-12T01:55:27.407 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1099876352 bytes 2023-12-12T01:55:27.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1129996288 bytes 2023-12-12T01:55:27.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1181410304 bytes 2023-12-12T01:55:27.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1242973184 bytes 2023-12-12T01:55:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_3 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_3 128M 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:28.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_3 2023-12-12T01:55:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:55:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:55:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:28.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2023-12-12T01:55:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T01:55:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:28.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-12T01:55:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:55:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-12T01:55:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_3': 2023-12-12T01:55:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:55:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:55:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:55:29.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 127e989e07d1 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.127e989e07d1 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:28 2023 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:28 2023 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:28 2023 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 489d4631-b2bd-47cc-9fa0-9d572c181d98 2023-12-12T01:55:29.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:55:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:55:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-12T01:55:29.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T01:55:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:55:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=14 2023-12-12T01:55:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:55:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:55:29.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 620544 bytes 2023-12-12T01:55:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 38242304 bytes 2023-12-12T01:55:29.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 108961792 bytes 2023-12-12T01:55:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 131500032 bytes 2023-12-12T01:55:29.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 178958336 bytes 2023-12-12T01:55:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 226071552 bytes 2023-12-12T01:55:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 239350784 bytes 2023-12-12T01:55:30.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 280729600 bytes 2023-12-12T01:55:30.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 303453184 bytes 2023-12-12T01:55:30.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 338581504 bytes 2023-12-12T01:55:30.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 360765440 bytes 2023-12-12T01:55:30.880 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 425605120 bytes 2023-12-12T01:55:31.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 447709184 bytes 2023-12-12T01:55:31.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 493753344 bytes 2023-12-12T01:55:31.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 549188608 bytes 2023-12-12T01:55:31.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 604590080 bytes 2023-12-12T01:55:31.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 638602240 bytes 2023-12-12T01:55:31.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 688393216 bytes 2023-12-12T01:55:32.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 766560256 bytes 2023-12-12T01:55:32.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 846827520 bytes 2023-12-12T01:55:32.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 852557824 bytes 2023-12-12T01:55:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 857993216 bytes 2023-12-12T01:55:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 924886016 bytes 2023-12-12T01:55:33.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1007323136 bytes 2023-12-12T01:55:33.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1012249600 bytes 2023-12-12T01:55:33.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1022521344 bytes 2023-12-12T01:55:33.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1097082880 bytes 2023-12-12T01:55:33.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1116769280 bytes 2023-12-12T01:55:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1150210048 bytes 2023-12-12T01:55:33.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1199262720 bytes 2023-12-12T01:55:34.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1234067456 bytes 2023-12-12T01:55:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1273067520 bytes 2023-12-12T01:55:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1312487424 bytes 2023-12-12T01:55:34.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1355401216 bytes 2023-12-12T01:55:34.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1380867072 bytes 2023-12-12T01:55:34.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1418110976 bytes 2023-12-12T01:55:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1433142272 bytes 2023-12-12T01:55:35.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1455651840 bytes 2023-12-12T01:55:35.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1518120960 bytes 2023-12-12T01:55:35.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1617776640 bytes 2023-12-12T01:55:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1670372352 bytes 2023-12-12T01:55:35.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1694397440 bytes 2023-12-12T01:55:36.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1720268800 bytes 2023-12-12T01:55:36.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1740299264 bytes 2023-12-12T01:55:36.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1789623296 bytes 2023-12-12T01:55:36.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1863788544 bytes 2023-12-12T01:55:36.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1886587904 bytes 2023-12-12T01:55:36.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1932684288 bytes 2023-12-12T01:55:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1982194688 bytes 2023-12-12T01:55:37.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2045739008 bytes 2023-12-12T01:55:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2046351360 bytes 2023-12-12T01:55:37.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2087669760 bytes 2023-12-12T01:55:37.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2144899072 bytes 2023-12-12T01:55:38.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2228237312 bytes 2023-12-12T01:55:38.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2228861952 bytes 2023-12-12T01:55:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2229485568 bytes 2023-12-12T01:55:38.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2305368064 bytes 2023-12-12T01:55:38.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2339014656 bytes 2023-12-12T01:55:38.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2351028224 bytes 2023-12-12T01:55:38.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2386487296 bytes 2023-12-12T01:55:39.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2463702016 bytes 2023-12-12T01:55:39.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2514163712 bytes 2023-12-12T01:55:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2578321408 bytes 2023-12-12T01:55:39.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2602447872 bytes 2023-12-12T01:55:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2665536512 bytes 2023-12-12T01:55:39.972 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5271942114890769) > weight (0.25) 2023-12-12T01:55:39.972 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T01:55:39.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T01:55:39.973 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-12T01:55:39.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:55:39.978+0000 7fa42d9c3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 45004) UID: 1000 2023-12-12T01:55:40.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2731192320 bytes 2023-12-12T01:55:40.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2767526912 bytes 2023-12-12T01:55:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2791763968 bytes 2023-12-12T01:55:40.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2816149504 bytes 2023-12-12T01:55:40.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2921910272 bytes 2023-12-12T01:55:40.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2959236096 bytes 2023-12-12T01:55:41.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3022617600 bytes 2023-12-12T01:55:41.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3100271616 bytes 2023-12-12T01:55:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3112800256 bytes 2023-12-12T01:55:41.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 3113433088 bytes 2023-12-12T01:55:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7600 IOs, 3128112128 bytes 2023-12-12T01:55:41.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7700 IOs, 3150247936 bytes 2023-12-12T01:55:41.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7800 IOs, 3171309568 bytes 2023-12-12T01:55:42.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7900 IOs, 3257337856 bytes 2023-12-12T01:55:42.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8000 IOs, 3364090880 bytes 2023-12-12T01:55:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8100 IOs, 3404962816 bytes 2023-12-12T01:55:42.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8200 IOs, 3429243904 bytes 2023-12-12T01:55:43.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_4 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:43.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_4 128M 2023-12-12T01:55:43.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:55:43.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:55:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_4 2023-12-12T01:55:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-12T01:55:43.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:55:43.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_4': 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1288437f076d 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1288437f076d 2023-12-12T01:55:44.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:43 2023 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:43 2023 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:43 2023 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 6fcc8ecf-2681-4d44-82de-245f7523ff13 2023-12-12T01:55:44.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T01:55:44.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:55:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T01:55:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:55:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:55:44.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 102079488 bytes 2023-12-12T01:55:44.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 125149184 bytes 2023-12-12T01:55:44.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 161696768 bytes 2023-12-12T01:55:44.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 172637184 bytes 2023-12-12T01:55:45.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 213805056 bytes 2023-12-12T01:55:45.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 264840192 bytes 2023-12-12T01:55:45.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 296875008 bytes 2023-12-12T01:55:45.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 322508800 bytes 2023-12-12T01:55:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 357646336 bytes 2023-12-12T01:55:45.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 397455360 bytes 2023-12-12T01:55:45.973 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T01:55:45.973 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:55:45.973 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T01:55:45.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T01:55:45.974 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T01:55:45.976 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T01:55:45.976 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-12T01:55:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 410478592 bytes 2023-12-12T01:55:46.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 474893312 bytes 2023-12-12T01:55:46.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 496813056 bytes 2023-12-12T01:55:46.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 497440768 bytes 2023-12-12T01:55:46.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 523961344 bytes 2023-12-12T01:55:47.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 587635712 bytes 2023-12-12T01:55:47.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 646494208 bytes 2023-12-12T01:55:47.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 681352192 bytes 2023-12-12T01:55:47.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 718673920 bytes 2023-12-12T01:55:47.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 752161792 bytes 2023-12-12T01:55:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 779510784 bytes 2023-12-12T01:55:48.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 837122048 bytes 2023-12-12T01:55:48.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 895020032 bytes 2023-12-12T01:55:48.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 912878592 bytes 2023-12-12T01:55:48.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 966930432 bytes 2023-12-12T01:55:48.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1047440384 bytes 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_5 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T01:55:49.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_5 128M 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:55:49.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_5 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:49.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T01:55:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:55:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-12T01:55:49.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:55:49.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-12T01:55:50.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_5': 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129ab8d3d384 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129ab8d3d384 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:49 2023 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:49 2023 2023-12-12T01:55:50.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:49 2023 2023-12-12T01:55:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:55:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:55:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c945b710-ea44-42b4-9616-704aab1766b4 2023-12-12T01:55:50.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:55:50.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:55:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-12T01:55:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:55:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:55:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T01:55:50.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:55:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:55:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:55:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:55:50.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 7320576 bytes 2023-12-12T01:55:50.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 32157696 bytes 2023-12-12T01:55:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 62418944 bytes 2023-12-12T01:55:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 135568384 bytes 2023-12-12T01:55:50.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 154099712 bytes 2023-12-12T01:55:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 176500736 bytes 2023-12-12T01:55:51.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 228590592 bytes 2023-12-12T01:55:51.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 290268160 bytes 2023-12-12T01:55:51.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 315591680 bytes 2023-12-12T01:55:51.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 338893824 bytes 2023-12-12T01:55:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 350561280 bytes 2023-12-12T01:55:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 396555264 bytes 2023-12-12T01:55:52.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 457365504 bytes 2023-12-12T01:55:52.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 519348224 bytes 2023-12-12T01:55:52.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 550529024 bytes 2023-12-12T01:55:52.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 613411840 bytes 2023-12-12T01:55:52.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 656073728 bytes 2023-12-12T01:55:52.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 676812800 bytes 2023-12-12T01:55:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 735970304 bytes 2023-12-12T01:55:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 816128000 bytes 2023-12-12T01:55:53.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 883704832 bytes 2023-12-12T01:55:53.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 943535104 bytes 2023-12-12T01:55:54.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 996125696 bytes 2023-12-12T01:55:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1075280896 bytes 2023-12-12T01:55:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1131646976 bytes 2023-12-12T01:55:54.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1163312128 bytes 2023-12-12T01:55:54.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1173186560 bytes 2023-12-12T01:55:54.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1199721472 bytes 2023-12-12T01:55:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1217994752 bytes 2023-12-12T01:55:55.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1249208320 bytes 2023-12-12T01:55:55.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1294704640 bytes 2023-12-12T01:55:55.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1402018816 bytes 2023-12-12T01:55:55.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1437873152 bytes 2023-12-12T01:55:55.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1479542784 bytes 2023-12-12T01:55:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1518139392 bytes 2023-12-12T01:55:56.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1595566080 bytes 2023-12-12T01:55:56.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1663716352 bytes 2023-12-12T01:55:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1688436736 bytes 2023-12-12T01:55:56.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1779091456 bytes 2023-12-12T01:55:57.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1804303360 bytes 2023-12-12T01:55:57.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1837327360 bytes 2023-12-12T01:55:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1875825664 bytes 2023-12-12T01:55:57.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1907505152 bytes 2023-12-12T01:55:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1939130368 bytes 2023-12-12T01:55:57.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1939728384 bytes 2023-12-12T01:55:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1954685952 bytes 2023-12-12T01:55:58.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2025239552 bytes 2023-12-12T01:55:58.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2063269888 bytes 2023-12-12T01:55:58.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2070721536 bytes 2023-12-12T01:55:58.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2124607488 bytes 2023-12-12T01:55:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2148640768 bytes 2023-12-12T01:55:58.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2168791040 bytes 2023-12-12T01:55:59.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2177770496 bytes 2023-12-12T01:55:59.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2205765632 bytes 2023-12-12T01:55:59.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2291578880 bytes 2023-12-12T01:55:59.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2327214080 bytes 2023-12-12T01:55:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2391096320 bytes 2023-12-12T01:55:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2428193792 bytes 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_6 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T01:56:00.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_6 128M 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_6 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T01:56:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:00.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-12T01:56:00.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:00.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-12T01:56:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_6': 2023-12-12T01:56:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a2b579e37c 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a2b579e37c 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:00 2023 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:00 2023 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:00 2023 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:01.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0e8bda32-db29-49f2-b777-53035364d63f 2023-12-12T01:56:01.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-12T01:56:01.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T01:56:01.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T01:56:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:01.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:01.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 23042048 bytes 2023-12-12T01:56:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 23653376 bytes 2023-12-12T01:56:01.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 92517376 bytes 2023-12-12T01:56:01.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 106916864 bytes 2023-12-12T01:56:01.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 125437952 bytes 2023-12-12T01:56:01.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 135553024 bytes 2023-12-12T01:56:01.980 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:01.985+0000 7fd7ab621640 -1 reset not still connected to 0x557474deb6c0 2023-12-12T01:56:02.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 174331904 bytes 2023-12-12T01:56:02.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 243196928 bytes 2023-12-12T01:56:02.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 296815616 bytes 2023-12-12T01:56:02.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 343016448 bytes 2023-12-12T01:56:02.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 364796928 bytes 2023-12-12T01:56:02.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 407939072 bytes 2023-12-12T01:56:03.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 428775424 bytes 2023-12-12T01:56:03.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 477494272 bytes 2023-12-12T01:56:03.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 525603840 bytes 2023-12-12T01:56:03.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 578590720 bytes 2023-12-12T01:56:04.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 631199744 bytes 2023-12-12T01:56:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 691749888 bytes 2023-12-12T01:56:04.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 719657984 bytes 2023-12-12T01:56:04.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 793042944 bytes 2023-12-12T01:56:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 900357120 bytes 2023-12-12T01:56:05.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 925927424 bytes 2023-12-12T01:56:05.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 966835200 bytes 2023-12-12T01:56:05.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 976175104 bytes 2023-12-12T01:56:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1045028864 bytes 2023-12-12T01:56:05.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1081692160 bytes 2023-12-12T01:56:05.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1142740992 bytes 2023-12-12T01:56:06.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1209609216 bytes 2023-12-12T01:56:06.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1230578688 bytes 2023-12-12T01:56:06.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1232241664 bytes 2023-12-12T01:56:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1259848704 bytes 2023-12-12T01:56:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1323914240 bytes 2023-12-12T01:56:06.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1375184896 bytes 2023-12-12T01:56:07.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1375798272 bytes 2023-12-12T01:56:07.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1452308480 bytes 2023-12-12T01:56:07.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1509534720 bytes 2023-12-12T01:56:07.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1543213056 bytes 2023-12-12T01:56:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1553679360 bytes 2023-12-12T01:56:07.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1586850816 bytes 2023-12-12T01:56:08.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1627896832 bytes 2023-12-12T01:56:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1674705920 bytes 2023-12-12T01:56:08.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1707486208 bytes 2023-12-12T01:56:08.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1759522816 bytes 2023-12-12T01:56:08.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1767814144 bytes 2023-12-12T01:56:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1810240512 bytes 2023-12-12T01:56:08.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1848536064 bytes 2023-12-12T01:56:09.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1904244736 bytes 2023-12-12T01:56:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1925778432 bytes 2023-12-12T01:56:09.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1977334784 bytes 2023-12-12T01:56:09.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1984557056 bytes 2023-12-12T01:56:09.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2038977536 bytes 2023-12-12T01:56:09.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2052749312 bytes 2023-12-12T01:56:10.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2086925312 bytes 2023-12-12T01:56:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_7 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:10.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_7 128M 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:10.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_7 2023-12-12T01:56:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2023-12-12T01:56:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T01:56:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:10.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-12T01:56:10.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:10.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6113322251836518) > weight (0.25) 2023-12-12T01:56:10.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7792392872527707) > weight (0.25) 2023-12-12T01:56:10.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6250784602566015) > weight (0.25) 2023-12-12T01:56:10.967 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6363199974151076) > weight (0.25) 2023-12-12T01:56:10.967 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-12T01:56:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_7': 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12acc45b0851 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12acc45b0851 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:11.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:10 2023 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:10 2023 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:10 2023 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: ce49be9c-8844-4083-8907-bd68b7a39229 2023-12-12T01:56:11.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T01:56:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T01:56:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 75399168 bytes 2023-12-12T01:56:11.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 89049088 bytes 2023-12-12T01:56:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 89649152 bytes 2023-12-12T01:56:11.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 131006464 bytes 2023-12-12T01:56:11.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 211172352 bytes 2023-12-12T01:56:12.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 250051584 bytes 2023-12-12T01:56:12.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 271555584 bytes 2023-12-12T01:56:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 296987648 bytes 2023-12-12T01:56:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 378516480 bytes 2023-12-12T01:56:12.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 400667648 bytes 2023-12-12T01:56:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 484545536 bytes 2023-12-12T01:56:13.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 520004608 bytes 2023-12-12T01:56:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 546274304 bytes 2023-12-12T01:56:13.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 566017024 bytes 2023-12-12T01:56:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 582522880 bytes 2023-12-12T01:56:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 653227008 bytes 2023-12-12T01:56:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 683774976 bytes 2023-12-12T01:56:13.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 728646656 bytes 2023-12-12T01:56:14.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 736364544 bytes 2023-12-12T01:56:14.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 754984960 bytes 2023-12-12T01:56:14.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 780709888 bytes 2023-12-12T01:56:14.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 796766208 bytes 2023-12-12T01:56:14.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 900905984 bytes 2023-12-12T01:56:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 935358464 bytes 2023-12-12T01:56:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 977114112 bytes 2023-12-12T01:56:15.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1048254464 bytes 2023-12-12T01:56:15.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1096423424 bytes 2023-12-12T01:56:15.771 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1174340608 bytes 2023-12-12T01:56:16.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1276237824 bytes 2023-12-12T01:56:16.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1299274752 bytes 2023-12-12T01:56:16.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1347054592 bytes 2023-12-12T01:56:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1379905536 bytes 2023-12-12T01:56:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1498735616 bytes 2023-12-12T01:56:16.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1529699328 bytes 2023-12-12T01:56:17.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1598563328 bytes 2023-12-12T01:56:17.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1655430144 bytes 2023-12-12T01:56:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1674390528 bytes 2023-12-12T01:56:17.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1766439936 bytes 2023-12-12T01:56:18.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1820730368 bytes 2023-12-12T01:56:18.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1858508800 bytes 2023-12-12T01:56:18.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1926189056 bytes 2023-12-12T01:56:18.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1955517440 bytes 2023-12-12T01:56:18.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1977643008 bytes 2023-12-12T01:56:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2034432000 bytes 2023-12-12T01:56:19.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2101627904 bytes 2023-12-12T01:56:19.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2110324736 bytes 2023-12-12T01:56:19.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2113549312 bytes 2023-12-12T01:56:19.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2145847296 bytes 2023-12-12T01:56:19.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2203521024 bytes 2023-12-12T01:56:19.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2227095552 bytes 2023-12-12T01:56:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2229386240 bytes 2023-12-12T01:56:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2247939072 bytes 2023-12-12T01:56:20.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2254517248 bytes 2023-12-12T01:56:20.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2306493440 bytes 2023-12-12T01:56:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2307119104 bytes 2023-12-12T01:56:20.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2383281152 bytes 2023-12-12T01:56:20.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2404249600 bytes 2023-12-12T01:56:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2427252736 bytes 2023-12-12T01:56:21.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2497354752 bytes 2023-12-12T01:56:21.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2522433536 bytes 2023-12-12T01:56:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2527408128 bytes 2023-12-12T01:56:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2605440000 bytes 2023-12-12T01:56:21.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2629890048 bytes 2023-12-12T01:56:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2664880128 bytes 2023-12-12T01:56:22.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2725981184 bytes 2023-12-12T01:56:22.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2782927872 bytes 2023-12-12T01:56:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2823792640 bytes 2023-12-12T01:56:22.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2836766720 bytes 2023-12-12T01:56:22.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2922047488 bytes 2023-12-12T01:56:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2980553728 bytes 2023-12-12T01:56:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 3041904640 bytes 2023-12-12T01:56:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3048595456 bytes 2023-12-12T01:56:23.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3130813440 bytes 2023-12-12T01:56:23.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3163969536 bytes 2023-12-12T01:56:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:24.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_8 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_8 128M 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:24.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:24.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_8 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:24.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T01:56:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:24.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-12T01:56:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:24.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_8': 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12b6a16b3ff4 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12b6a16b3ff4 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:24 2023 2023-12-12T01:56:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:24 2023 2023-12-12T01:56:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:24 2023 2023-12-12T01:56:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 342b1cab-88ee-458a-94d9-2ad59864a055 2023-12-12T01:56:25.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T01:56:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T01:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:25.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 67799040 bytes 2023-12-12T01:56:25.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 157975552 bytes 2023-12-12T01:56:25.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 186038272 bytes 2023-12-12T01:56:25.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 211028992 bytes 2023-12-12T01:56:25.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 226204672 bytes 2023-12-12T01:56:26.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 245299200 bytes 2023-12-12T01:56:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 329074688 bytes 2023-12-12T01:56:26.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 367827968 bytes 2023-12-12T01:56:26.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 414340096 bytes 2023-12-12T01:56:26.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 458474496 bytes 2023-12-12T01:56:27.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 550281216 bytes 2023-12-12T01:56:27.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 690807808 bytes 2023-12-12T01:56:27.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 754285568 bytes 2023-12-12T01:56:27.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 805047296 bytes 2023-12-12T01:56:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 837380096 bytes 2023-12-12T01:56:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 881677312 bytes 2023-12-12T01:56:28.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 901648384 bytes 2023-12-12T01:56:28.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 964018176 bytes 2023-12-12T01:56:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1003331584 bytes 2023-12-12T01:56:29.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1090909184 bytes 2023-12-12T01:56:29.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1107171328 bytes 2023-12-12T01:56:29.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1151868928 bytes 2023-12-12T01:56:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1168839680 bytes 2023-12-12T01:56:29.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1169467392 bytes 2023-12-12T01:56:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1225150464 bytes 2023-12-12T01:56:30.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1311229952 bytes 2023-12-12T01:56:30.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1311849472 bytes 2023-12-12T01:56:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1335329792 bytes 2023-12-12T01:56:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1370023936 bytes 2023-12-12T01:56:30.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1442384896 bytes 2023-12-12T01:56:31.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1517583360 bytes 2023-12-12T01:56:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1574308864 bytes 2023-12-12T01:56:31.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1574924288 bytes 2023-12-12T01:56:31.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1601220608 bytes 2023-12-12T01:56:31.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1665436672 bytes 2023-12-12T01:56:31.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1693395968 bytes 2023-12-12T01:56:31.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1728504832 bytes 2023-12-12T01:56:32.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1746476032 bytes 2023-12-12T01:56:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1768824832 bytes 2023-12-12T01:56:32.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1879827456 bytes 2023-12-12T01:56:32.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1910243328 bytes 2023-12-12T01:56:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1919290368 bytes 2023-12-12T01:56:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1945116672 bytes 2023-12-12T01:56:33.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2000183296 bytes 2023-12-12T01:56:33.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2044595200 bytes 2023-12-12T01:56:33.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2089716736 bytes 2023-12-12T01:56:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2114356224 bytes 2023-12-12T01:56:33.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2165877760 bytes 2023-12-12T01:56:34.045 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_9 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:34.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_9 128M 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:34.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:34.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_9 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:34.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T01:56:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-12T01:56:35.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-12T01:56:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_9': 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12bfefefa2c1 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12bfefefa2c1 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:34 2023 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:34 2023 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:34 2023 2023-12-12T01:56:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 55c1b0cd-97fd-4de6-bd65-a62b4f88d578 2023-12-12T01:56:35.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:35.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-12T01:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T01:56:35.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:35.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 31208448 bytes 2023-12-12T01:56:35.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 69085184 bytes 2023-12-12T01:56:35.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 111145984 bytes 2023-12-12T01:56:35.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 147428352 bytes 2023-12-12T01:56:36.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 215370752 bytes 2023-12-12T01:56:36.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 275885056 bytes 2023-12-12T01:56:36.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 290429952 bytes 2023-12-12T01:56:36.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 321402880 bytes 2023-12-12T01:56:36.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 401460224 bytes 2023-12-12T01:56:36.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 418147328 bytes 2023-12-12T01:56:37.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 445243392 bytes 2023-12-12T01:56:37.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 490254336 bytes 2023-12-12T01:56:37.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 493516800 bytes 2023-12-12T01:56:37.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 544032768 bytes 2023-12-12T01:56:37.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 644211712 bytes 2023-12-12T01:56:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 664495104 bytes 2023-12-12T01:56:38.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 700592128 bytes 2023-12-12T01:56:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 722193408 bytes 2023-12-12T01:56:38.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 735831040 bytes 2023-12-12T01:56:38.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 774609920 bytes 2023-12-12T01:56:38.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 783700992 bytes 2023-12-12T01:56:38.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 858660864 bytes 2023-12-12T01:56:38.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 866053120 bytes 2023-12-12T01:56:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 909787136 bytes 2023-12-12T01:56:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 968554496 bytes 2023-12-12T01:56:39.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 994084864 bytes 2023-12-12T01:56:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1026402304 bytes 2023-12-12T01:56:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1134325760 bytes 2023-12-12T01:56:40.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1184012288 bytes 2023-12-12T01:56:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1216986112 bytes 2023-12-12T01:56:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1257788416 bytes 2023-12-12T01:56:40.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1311136768 bytes 2023-12-12T01:56:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1379822592 bytes 2023-12-12T01:56:41.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1449421824 bytes 2023-12-12T01:56:41.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1488030720 bytes 2023-12-12T01:56:41.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1514686464 bytes 2023-12-12T01:56:41.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1531948032 bytes 2023-12-12T01:56:41.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1576722432 bytes 2023-12-12T01:56:41.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1598603264 bytes 2023-12-12T01:56:42.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1638864896 bytes 2023-12-12T01:56:42.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1640207360 bytes 2023-12-12T01:56:42.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1685093376 bytes 2023-12-12T01:56:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1789355008 bytes 2023-12-12T01:56:42.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1895170048 bytes 2023-12-12T01:56:43.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1895792640 bytes 2023-12-12T01:56:43.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1935511552 bytes 2023-12-12T01:56:43.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2050015232 bytes 2023-12-12T01:56:43.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2050635776 bytes 2023-12-12T01:56:43.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2093775872 bytes 2023-12-12T01:56:43.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2136242176 bytes 2023-12-12T01:56:44.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2146194432 bytes 2023-12-12T01:56:44.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2183200768 bytes 2023-12-12T01:56:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2210203648 bytes 2023-12-12T01:56:44.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2310327296 bytes 2023-12-12T01:56:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2370784256 bytes 2023-12-12T01:56:45.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2385618944 bytes 2023-12-12T01:56:45.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_10 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_10 128M 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_10 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:45.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T01:56:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:45.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-12T01:56:46.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:46.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_10': 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12c88199947e 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12c88199947e 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:45 2023 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:45 2023 2023-12-12T01:56:46.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:45 2023 2023-12-12T01:56:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 73a5838f-6323-4afc-bd11-554e7ecf7c56 2023-12-12T01:56:46.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T01:56:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=7 2023-12-12T01:56:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:46.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 13659136 bytes 2023-12-12T01:56:46.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 59880448 bytes 2023-12-12T01:56:46.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 82839552 bytes 2023-12-12T01:56:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 83441664 bytes 2023-12-12T01:56:46.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 101215232 bytes 2023-12-12T01:56:46.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 148424704 bytes 2023-12-12T01:56:47.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 167890944 bytes 2023-12-12T01:56:47.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 195649536 bytes 2023-12-12T01:56:47.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 236005376 bytes 2023-12-12T01:56:47.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 313144320 bytes 2023-12-12T01:56:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 365544448 bytes 2023-12-12T01:56:48.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 421033984 bytes 2023-12-12T01:56:48.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 436791296 bytes 2023-12-12T01:56:48.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 475881472 bytes 2023-12-12T01:56:48.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 523473920 bytes 2023-12-12T01:56:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 608048128 bytes 2023-12-12T01:56:49.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 686024704 bytes 2023-12-12T01:56:49.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 696326144 bytes 2023-12-12T01:56:49.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 772091904 bytes 2023-12-12T01:56:49.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 877945856 bytes 2023-12-12T01:56:50.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 983305216 bytes 2023-12-12T01:56:50.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1028080640 bytes 2023-12-12T01:56:50.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1042384896 bytes 2023-12-12T01:56:50.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1066867712 bytes 2023-12-12T01:56:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1100283904 bytes 2023-12-12T01:56:50.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1130788864 bytes 2023-12-12T01:56:51.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1176616960 bytes 2023-12-12T01:56:51.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1195933696 bytes 2023-12-12T01:56:51.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1210153984 bytes 2023-12-12T01:56:51.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1261808640 bytes 2023-12-12T01:56:51.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1262434304 bytes 2023-12-12T01:56:51.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1295331328 bytes 2023-12-12T01:56:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1317612544 bytes 2023-12-12T01:56:52.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1372894208 bytes 2023-12-12T01:56:52.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1428755456 bytes 2023-12-12T01:56:52.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1509126144 bytes 2023-12-12T01:56:52.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1605342208 bytes 2023-12-12T01:56:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1617695744 bytes 2023-12-12T01:56:53.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1625404416 bytes 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_11 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_11 128M 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:56:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:56:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:56:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:56:53.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_11 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T01:56:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:56:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2023-12-12T01:56:54.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:56:54.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_11': 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12d28abac57c 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12d28abac57c 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:56:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:53 2023 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:53 2023 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:53 2023 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: dfbf3fd7-eb15-4e16-a0ac-43c2a8e3e631 2023-12-12T01:56:54.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_11 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T01:56:54.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:56:54.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T01:56:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:56:54.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:56:54.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 53758976 bytes 2023-12-12T01:56:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 77818880 bytes 2023-12-12T01:56:54.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 114769920 bytes 2023-12-12T01:56:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 121054208 bytes 2023-12-12T01:56:55.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 183518208 bytes 2023-12-12T01:56:55.118 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:55.121+0000 7fd7ab621640 -1 reset not still connected to 0x557474a6f2b0 2023-12-12T01:56:55.118 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:55.121+0000 7fd7ab621640 -1 reset not still connected to 0x557474deb6c0 2023-12-12T01:56:55.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 245972992 bytes 2023-12-12T01:56:55.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 298918912 bytes 2023-12-12T01:56:55.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 310511616 bytes 2023-12-12T01:56:55.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 353364992 bytes 2023-12-12T01:56:55.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 439118848 bytes 2023-12-12T01:56:56.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 447698944 bytes 2023-12-12T01:56:56.214 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:56.213+0000 7fd7ab621640 -1 reset not still connected to 0x557474a6e8f0 2023-12-12T01:56:56.214 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:56.213+0000 7fd7ab621640 -1 reset not still connected to 0x557474a6f040 2023-12-12T01:56:56.214 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:56:56.213+0000 7fd7ab621640 -1 reset not still connected to 0x557474a6f2b0 2023-12-12T01:56:56.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 507266048 bytes 2023-12-12T01:56:56.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 581905408 bytes 2023-12-12T01:56:56.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 613044224 bytes 2023-12-12T01:56:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 622372864 bytes 2023-12-12T01:56:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 623014912 bytes 2023-12-12T01:56:57.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 651466752 bytes 2023-12-12T01:56:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 676573184 bytes 2023-12-12T01:56:57.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 803129344 bytes 2023-12-12T01:56:57.711 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 847516672 bytes 2023-12-12T01:56:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 854320128 bytes 2023-12-12T01:56:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 854922240 bytes 2023-12-12T01:56:58.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 899981312 bytes 2023-12-12T01:56:58.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 943125504 bytes 2023-12-12T01:56:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 968547328 bytes 2023-12-12T01:56:58.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1006628864 bytes 2023-12-12T01:56:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1071351808 bytes 2023-12-12T01:56:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1162233856 bytes 2023-12-12T01:56:59.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1213006848 bytes 2023-12-12T01:56:59.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1231078400 bytes 2023-12-12T01:56:59.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1235369984 bytes 2023-12-12T01:56:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1258012672 bytes 2023-12-12T01:56:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1315673088 bytes 2023-12-12T01:56:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1318259712 bytes 2023-12-12T01:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:00.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_12 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_12 128M 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:00.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_12 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:00.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2023-12-12T01:57:00.977 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8963688018179697) > weight (0.25) 2023-12-12T01:57:00.977 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T01:57:00.977 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T01:57:00.977 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T01:57:00.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T01:57:00.981+0000 7fc6a8be5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 48782) UID: 1000 2023-12-12T01:57:00.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:00.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_12': 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12dbb06df4c4 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12dbb06df4c4 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:00 2023 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:00 2023 2023-12-12T01:57:01.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:00 2023 2023-12-12T01:57:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: ddfde94a-a349-4e2a-ae4d-72258f5591a6 2023-12-12T01:57:01.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_12 2023-12-12T01:57:01.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T01:57:01.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T01:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:01.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 56579072 bytes 2023-12-12T01:57:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 100008960 bytes 2023-12-12T01:57:01.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 162017280 bytes 2023-12-12T01:57:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 214449152 bytes 2023-12-12T01:57:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 217286656 bytes 2023-12-12T01:57:02.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 245196800 bytes 2023-12-12T01:57:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 284656640 bytes 2023-12-12T01:57:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 298208256 bytes 2023-12-12T01:57:02.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 320318464 bytes 2023-12-12T01:57:02.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 365943808 bytes 2023-12-12T01:57:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 393196544 bytes 2023-12-12T01:57:03.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 457995264 bytes 2023-12-12T01:57:03.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 469581824 bytes 2023-12-12T01:57:03.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 470210560 bytes 2023-12-12T01:57:03.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 507453440 bytes 2023-12-12T01:57:03.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 518658048 bytes 2023-12-12T01:57:03.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 565758976 bytes 2023-12-12T01:57:04.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 580549632 bytes 2023-12-12T01:57:04.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 640314368 bytes 2023-12-12T01:57:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 646805504 bytes 2023-12-12T01:57:04.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 710406144 bytes 2023-12-12T01:57:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 763193344 bytes 2023-12-12T01:57:04.978 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T01:57:04.978 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:57:04.979 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T01:57:04.979 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T01:57:04.979 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T01:57:04.982 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T01:57:04.982 INFO:tasks.rbd_mirror_thrash:waiting for 80 secs before thrashing 2023-12-12T01:57:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 805620736 bytes 2023-12-12T01:57:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 850734080 bytes 2023-12-12T01:57:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 897506304 bytes 2023-12-12T01:57:05.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 925625344 bytes 2023-12-12T01:57:05.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1001091072 bytes 2023-12-12T01:57:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1041599488 bytes 2023-12-12T01:57:06.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1154327552 bytes 2023-12-12T01:57:06.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1154960384 bytes 2023-12-12T01:57:06.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1164142592 bytes 2023-12-12T01:57:06.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1222497280 bytes 2023-12-12T01:57:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1258169344 bytes 2023-12-12T01:57:07.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1280093184 bytes 2023-12-12T01:57:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1349998592 bytes 2023-12-12T01:57:07.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1396977664 bytes 2023-12-12T01:57:07.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1427661824 bytes 2023-12-12T01:57:07.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1494818816 bytes 2023-12-12T01:57:08.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1533434880 bytes 2023-12-12T01:57:08.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1541597184 bytes 2023-12-12T01:57:08.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1573556224 bytes 2023-12-12T01:57:08.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1612473344 bytes 2023-12-12T01:57:08.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1649010688 bytes 2023-12-12T01:57:08.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1684639744 bytes 2023-12-12T01:57:08.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1723604992 bytes 2023-12-12T01:57:09.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1749870592 bytes 2023-12-12T01:57:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:09.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_13 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_13 128M 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:09.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:09.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_13 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T01:57:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:09.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2023-12-12T01:57:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_13': 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12ea15ccad76 2023-12-12T01:57:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12ea15ccad76 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:09 2023 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:09 2023 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:09 2023 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 6c841662-f2e7-4c7e-8c09-62dd32e1eff8 2023-12-12T01:57:10.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_13 2023-12-12T01:57:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:10.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T01:57:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:10.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=14 2023-12-12T01:57:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 8576000 bytes 2023-12-12T01:57:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 60950528 bytes 2023-12-12T01:57:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 105451520 bytes 2023-12-12T01:57:10.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 167924736 bytes 2023-12-12T01:57:11.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 237460480 bytes 2023-12-12T01:57:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 275673088 bytes 2023-12-12T01:57:11.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 336202752 bytes 2023-12-12T01:57:11.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 357422080 bytes 2023-12-12T01:57:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 388557824 bytes 2023-12-12T01:57:11.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 443484160 bytes 2023-12-12T01:57:12.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 488957952 bytes 2023-12-12T01:57:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 530644992 bytes 2023-12-12T01:57:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 575002624 bytes 2023-12-12T01:57:12.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 596606976 bytes 2023-12-12T01:57:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 683316224 bytes 2023-12-12T01:57:13.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 729703424 bytes 2023-12-12T01:57:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 786294784 bytes 2023-12-12T01:57:13.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 823388160 bytes 2023-12-12T01:57:13.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 867215360 bytes 2023-12-12T01:57:13.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 878394368 bytes 2023-12-12T01:57:13.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 907096064 bytes 2023-12-12T01:57:14.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 977454080 bytes 2023-12-12T01:57:14.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 979082240 bytes 2023-12-12T01:57:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1022182400 bytes 2023-12-12T01:57:14.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1047786496 bytes 2023-12-12T01:57:14.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1049785344 bytes 2023-12-12T01:57:14.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1050398720 bytes 2023-12-12T01:57:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1122186240 bytes 2023-12-12T01:57:14.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3428686222076881) > weight (0.25) 2023-12-12T01:57:14.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.346772967049396) > weight (0.25) 2023-12-12T01:57:14.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.8403838385193079) > weight (0.25) 2023-12-12T01:57:14.968 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5237937578217727) > weight (0.25) 2023-12-12T01:57:14.968 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-12-12T01:57:15.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1150253056 bytes 2023-12-12T01:57:15.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1213020160 bytes 2023-12-12T01:57:15.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1213640704 bytes 2023-12-12T01:57:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1237869568 bytes 2023-12-12T01:57:15.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1298915328 bytes 2023-12-12T01:57:15.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1316918272 bytes 2023-12-12T01:57:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1383617536 bytes 2023-12-12T01:57:16.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1456542720 bytes 2023-12-12T01:57:16.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1461019648 bytes 2023-12-12T01:57:16.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1527221248 bytes 2023-12-12T01:57:16.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1572417536 bytes 2023-12-12T01:57:17.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1638155264 bytes 2023-12-12T01:57:17.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1659657216 bytes 2023-12-12T01:57:17.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1684428800 bytes 2023-12-12T01:57:17.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1741164544 bytes 2023-12-12T01:57:17.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1818058752 bytes 2023-12-12T01:57:18.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1851554816 bytes 2023-12-12T01:57:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1911117824 bytes 2023-12-12T01:57:18.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1954287616 bytes 2023-12-12T01:57:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1983874048 bytes 2023-12-12T01:57:18.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2061334528 bytes 2023-12-12T01:57:18.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2084904960 bytes 2023-12-12T01:57:19.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2106312704 bytes 2023-12-12T01:57:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2151177216 bytes 2023-12-12T01:57:19.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2176773120 bytes 2023-12-12T01:57:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2222371840 bytes 2023-12-12T01:57:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2237353984 bytes 2023-12-12T01:57:19.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2237960192 bytes 2023-12-12T01:57:20.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2324061184 bytes 2023-12-12T01:57:20.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2362106880 bytes 2023-12-12T01:57:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2455126016 bytes 2023-12-12T01:57:20.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2466088960 bytes 2023-12-12T01:57:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2526006272 bytes 2023-12-12T01:57:21.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2570259456 bytes 2023-12-12T01:57:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2610548736 bytes 2023-12-12T01:57:21.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2672591872 bytes 2023-12-12T01:57:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2710598656 bytes 2023-12-12T01:57:21.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2769600512 bytes 2023-12-12T01:57:22.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2800471040 bytes 2023-12-12T01:57:22.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2889170944 bytes 2023-12-12T01:57:22.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2906981376 bytes 2023-12-12T01:57:22.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2987446272 bytes 2023-12-12T01:57:22.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 3026756608 bytes 2023-12-12T01:57:23.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3088093184 bytes 2023-12-12T01:57:23.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3099948032 bytes 2023-12-12T01:57:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3109380096 bytes 2023-12-12T01:57:23.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 3163276288 bytes 2023-12-12T01:57:23.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7600 IOs, 3199060992 bytes 2023-12-12T01:57:23.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7700 IOs, 3205742592 bytes 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_14 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_14 128M 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:24.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_14 2023-12-12T01:57:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:24.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2023-12-12T01:57:25.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:25.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2023-12-12T01:57:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_14': 2023-12-12T01:57:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:25.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12f4b2ec2988 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12f4b2ec2988 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:24 2023 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:24 2023 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:24 2023 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c9956d0a-2c28-4015-9421-074ee875e996 2023-12-12T01:57:25.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_14 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T01:57:25.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:25.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=7 2023-12-12T01:57:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:25.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 41928704 bytes 2023-12-12T01:57:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 80987136 bytes 2023-12-12T01:57:25.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 112257024 bytes 2023-12-12T01:57:26.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 173120512 bytes 2023-12-12T01:57:26.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 215647232 bytes 2023-12-12T01:57:26.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 229980160 bytes 2023-12-12T01:57:26.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 283827200 bytes 2023-12-12T01:57:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 335118336 bytes 2023-12-12T01:57:27.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 422190080 bytes 2023-12-12T01:57:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 435540992 bytes 2023-12-12T01:57:27.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 500809728 bytes 2023-12-12T01:57:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 553984000 bytes 2023-12-12T01:57:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 586281984 bytes 2023-12-12T01:57:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 627818496 bytes 2023-12-12T01:57:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 702167040 bytes 2023-12-12T01:57:28.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 731832320 bytes 2023-12-12T01:57:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 795941888 bytes 2023-12-12T01:57:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 845672448 bytes 2023-12-12T01:57:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 855038976 bytes 2023-12-12T01:57:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 888067072 bytes 2023-12-12T01:57:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 908258304 bytes 2023-12-12T01:57:29.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 973146112 bytes 2023-12-12T01:57:29.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1020809216 bytes 2023-12-12T01:57:29.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1039978496 bytes 2023-12-12T01:57:29.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1040586752 bytes 2023-12-12T01:57:29.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1059444736 bytes 2023-12-12T01:57:30.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1112783872 bytes 2023-12-12T01:57:30.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1170495488 bytes 2023-12-12T01:57:30.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1173604352 bytes 2023-12-12T01:57:30.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1207568384 bytes 2023-12-12T01:57:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1301684224 bytes 2023-12-12T01:57:31.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1323445248 bytes 2023-12-12T01:57:31.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1334201344 bytes 2023-12-12T01:57:31.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1334800384 bytes 2023-12-12T01:57:31.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1380920320 bytes 2023-12-12T01:57:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1450711040 bytes 2023-12-12T01:57:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1517142016 bytes 2023-12-12T01:57:32.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1533924352 bytes 2023-12-12T01:57:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:32.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_15 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_15 128M 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:32.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:32.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_15 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2023-12-12T01:57:32.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T01:57:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:32.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2023-12-12T01:57:33.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:33.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2023-12-12T01:57:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_15': 2023-12-12T01:57:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:33.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12feb733789 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12feb733789 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 09d8c9b1-86e1-481e-8a12-992f2d80f06d 2023-12-12T01:57:33.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_15 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T01:57:33.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:57:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:33.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 46420992 bytes 2023-12-12T01:57:33.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 133181440 bytes 2023-12-12T01:57:33.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 197020672 bytes 2023-12-12T01:57:34.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 248764416 bytes 2023-12-12T01:57:34.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 253465600 bytes 2023-12-12T01:57:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 280147968 bytes 2023-12-12T01:57:34.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 347845632 bytes 2023-12-12T01:57:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 397563904 bytes 2023-12-12T01:57:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 431928320 bytes 2023-12-12T01:57:35.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 508892160 bytes 2023-12-12T01:57:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 509648896 bytes 2023-12-12T01:57:35.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 528630784 bytes 2023-12-12T01:57:35.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 529460224 bytes 2023-12-12T01:57:35.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 581892096 bytes 2023-12-12T01:57:35.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 621302784 bytes 2023-12-12T01:57:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 668666880 bytes 2023-12-12T01:57:36.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 681409536 bytes 2023-12-12T01:57:36.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 746663936 bytes 2023-12-12T01:57:36.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 805136384 bytes 2023-12-12T01:57:36.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 819268608 bytes 2023-12-12T01:57:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 881052672 bytes 2023-12-12T01:57:37.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 924360704 bytes 2023-12-12T01:57:37.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 936070144 bytes 2023-12-12T01:57:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 936700928 bytes 2023-12-12T01:57:37.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 996814848 bytes 2023-12-12T01:57:37.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1023883264 bytes 2023-12-12T01:57:38.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1107307520 bytes 2023-12-12T01:57:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1151428608 bytes 2023-12-12T01:57:38.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1198658560 bytes 2023-12-12T01:57:38.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1256011776 bytes 2023-12-12T01:57:38.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1301624832 bytes 2023-12-12T01:57:39.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1339954176 bytes 2023-12-12T01:57:39.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1377459200 bytes 2023-12-12T01:57:39.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1384324096 bytes 2023-12-12T01:57:39.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1426136064 bytes 2023-12-12T01:57:39.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1454542848 bytes 2023-12-12T01:57:39.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1505899520 bytes 2023-12-12T01:57:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1506524160 bytes 2023-12-12T01:57:40.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1540720640 bytes 2023-12-12T01:57:40.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1598496768 bytes 2023-12-12T01:57:40.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1626391552 bytes 2023-12-12T01:57:40.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1721589760 bytes 2023-12-12T01:57:40.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1739768832 bytes 2023-12-12T01:57:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1764850688 bytes 2023-12-12T01:57:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1773654016 bytes 2023-12-12T01:57:41.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1774726144 bytes 2023-12-12T01:57:41.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1818286080 bytes 2023-12-12T01:57:41.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1854123008 bytes 2023-12-12T01:57:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1885172736 bytes 2023-12-12T01:57:42.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1915896832 bytes 2023-12-12T01:57:42.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1998260224 bytes 2023-12-12T01:57:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2060164096 bytes 2023-12-12T01:57:42.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2122311680 bytes 2023-12-12T01:57:42.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2137609216 bytes 2023-12-12T01:57:43.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2178974720 bytes 2023-12-12T01:57:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_16 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_16 128M 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:43.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_16 2023-12-12T01:57:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:43.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T01:57:43.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2023-12-12T01:57:44.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:44.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_16': 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1307a955f7a 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1307a955f7a 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:44.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:43 2023 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:43 2023 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:43 2023 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: cf75936b-449b-47ab-8ea6-3647877a3e66 2023-12-12T01:57:44.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_16 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T01:57:44.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:57:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:44.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 84193280 bytes 2023-12-12T01:57:44.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 150841344 bytes 2023-12-12T01:57:44.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 193519616 bytes 2023-12-12T01:57:44.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 236798976 bytes 2023-12-12T01:57:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 287478784 bytes 2023-12-12T01:57:45.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 355516416 bytes 2023-12-12T01:57:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 368873472 bytes 2023-12-12T01:57:45.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 421069824 bytes 2023-12-12T01:57:45.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 447843328 bytes 2023-12-12T01:57:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 463344640 bytes 2023-12-12T01:57:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 476726272 bytes 2023-12-12T01:57:46.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 565380096 bytes 2023-12-12T01:57:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 585588736 bytes 2023-12-12T01:57:46.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 621955072 bytes 2023-12-12T01:57:46.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 642309120 bytes 2023-12-12T01:57:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 692388864 bytes 2023-12-12T01:57:47.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 754091008 bytes 2023-12-12T01:57:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 807239680 bytes 2023-12-12T01:57:47.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 847329280 bytes 2023-12-12T01:57:47.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 897160192 bytes 2023-12-12T01:57:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 983527424 bytes 2023-12-12T01:57:48.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1023973376 bytes 2023-12-12T01:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1066515456 bytes 2023-12-12T01:57:48.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1067125760 bytes 2023-12-12T01:57:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1121950720 bytes 2023-12-12T01:57:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1227542528 bytes 2023-12-12T01:57:49.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1303879680 bytes 2023-12-12T01:57:49.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1351305216 bytes 2023-12-12T01:57:49.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1380267008 bytes 2023-12-12T01:57:50.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1482061824 bytes 2023-12-12T01:57:50.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1537313792 bytes 2023-12-12T01:57:50.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1542179840 bytes 2023-12-12T01:57:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1554329600 bytes 2023-12-12T01:57:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1637944320 bytes 2023-12-12T01:57:51.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1685002240 bytes 2023-12-12T01:57:51.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1724910592 bytes 2023-12-12T01:57:51.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1816852480 bytes 2023-12-12T01:57:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1851076608 bytes 2023-12-12T01:57:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1899851776 bytes 2023-12-12T01:57:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1901139968 bytes 2023-12-12T01:57:52.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1981858816 bytes 2023-12-12T01:57:52.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 2028252160 bytes 2023-12-12T01:57:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 2064414720 bytes 2023-12-12T01:57:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2125016064 bytes 2023-12-12T01:57:53.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2157919232 bytes 2023-12-12T01:57:53.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2193730560 bytes 2023-12-12T01:57:53.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2274575360 bytes 2023-12-12T01:57:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2316831744 bytes 2023-12-12T01:57:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:57:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:57:54.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_17 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_17 128M 2023-12-12T01:57:54.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:57:54.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_17 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T01:57:54.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:57:54.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2023-12-12T01:57:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:57:55.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2023-12-12T01:57:55.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_17': 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 130faf1afe43 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.130faf1afe43 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:54 2023 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:54 2023 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:54 2023 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:57:55.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:57:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d06e3eb6-22a3-4ffe-88ca-04ba98f63457 2023-12-12T01:57:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_17 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T01:57:55.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:57:55.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T01:57:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:57:55.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:57:55.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 93938688 bytes 2023-12-12T01:57:55.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 119158784 bytes 2023-12-12T01:57:55.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 186577920 bytes 2023-12-12T01:57:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 253566976 bytes 2023-12-12T01:57:56.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 295366656 bytes 2023-12-12T01:57:56.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 305752064 bytes 2023-12-12T01:57:56.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 326016000 bytes 2023-12-12T01:57:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 369124352 bytes 2023-12-12T01:57:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 404203520 bytes 2023-12-12T01:57:57.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 447737856 bytes 2023-12-12T01:57:57.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 507336704 bytes 2023-12-12T01:57:57.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 515706880 bytes 2023-12-12T01:57:57.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 578249728 bytes 2023-12-12T01:57:57.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 641790976 bytes 2023-12-12T01:57:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 697142272 bytes 2023-12-12T01:57:58.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 717444096 bytes 2023-12-12T01:57:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 777376768 bytes 2023-12-12T01:57:58.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 808148992 bytes 2023-12-12T01:57:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 842000384 bytes 2023-12-12T01:57:58.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 842602496 bytes 2023-12-12T01:57:59.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 918238208 bytes 2023-12-12T01:57:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 938551296 bytes 2023-12-12T01:57:59.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 939151360 bytes 2023-12-12T01:57:59.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 985551872 bytes 2023-12-12T01:57:59.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1055650816 bytes 2023-12-12T01:57:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1102071808 bytes 2023-12-12T01:58:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1185218560 bytes 2023-12-12T01:58:00.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1290224640 bytes 2023-12-12T01:58:00.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1317497856 bytes 2023-12-12T01:58:00.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1323275264 bytes 2023-12-12T01:58:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1339259904 bytes 2023-12-12T01:58:01.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1380843520 bytes 2023-12-12T01:58:01.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1401299968 bytes 2023-12-12T01:58:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1507554304 bytes 2023-12-12T01:58:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1574241280 bytes 2023-12-12T01:58:01.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1609605120 bytes 2023-12-12T01:58:02.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1652877312 bytes 2023-12-12T01:58:02.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1682791424 bytes 2023-12-12T01:58:02.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1737171968 bytes 2023-12-12T01:58:02.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1750687744 bytes 2023-12-12T01:58:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1751306240 bytes 2023-12-12T01:58:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1768426496 bytes 2023-12-12T01:58:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1828405248 bytes 2023-12-12T01:58:03.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1829011456 bytes 2023-12-12T01:58:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1906501632 bytes 2023-12-12T01:58:03.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1929128960 bytes 2023-12-12T01:58:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1945401344 bytes 2023-12-12T01:58:03.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2005690368 bytes 2023-12-12T01:58:04.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2066828288 bytes 2023-12-12T01:58:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2082826240 bytes 2023-12-12T01:58:04.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2133743616 bytes 2023-12-12T01:58:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2135028736 bytes 2023-12-12T01:58:04.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2163697664 bytes 2023-12-12T01:58:04.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2164330496 bytes 2023-12-12T01:58:05.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2229844992 bytes 2023-12-12T01:58:05.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2257251328 bytes 2023-12-12T01:58:05.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2270291968 bytes 2023-12-12T01:58:05.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2343123968 bytes 2023-12-12T01:58:05.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2370512896 bytes 2023-12-12T01:58:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2412169216 bytes 2023-12-12T01:58:06.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2415141888 bytes 2023-12-12T01:58:06.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2430877696 bytes 2023-12-12T01:58:06.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2485299200 bytes 2023-12-12T01:58:06.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2497894400 bytes 2023-12-12T01:58:06.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2531130368 bytes 2023-12-12T01:58:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2569818112 bytes 2023-12-12T01:58:07.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2601322496 bytes 2023-12-12T01:58:07.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2665919488 bytes 2023-12-12T01:58:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2731911168 bytes 2023-12-12T01:58:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2774114304 bytes 2023-12-12T01:58:07.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2808134656 bytes 2023-12-12T01:58:08.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_18 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:08.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_18 128M 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:08.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_18 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:08.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T01:58:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:08.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2023-12-12T01:58:09.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:58:09.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2023-12-12T01:58:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_18': 2023-12-12T01:58:09.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 131760fefde3 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.131760fefde3 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:08 2023 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:08 2023 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:08 2023 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 2db15675-ec89-40f1-b944-460662d253f6 2023-12-12T01:58:09.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_18 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T01:58:09.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T01:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:58:09.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:58:09.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 47356928 bytes 2023-12-12T01:58:09.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 75681792 bytes 2023-12-12T01:58:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 104931328 bytes 2023-12-12T01:58:09.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 116804608 bytes 2023-12-12T01:58:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 144904192 bytes 2023-12-12T01:58:10.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 191173632 bytes 2023-12-12T01:58:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 227156992 bytes 2023-12-12T01:58:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 267735040 bytes 2023-12-12T01:58:10.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 304163840 bytes 2023-12-12T01:58:10.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 337528832 bytes 2023-12-12T01:58:11.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 486422528 bytes 2023-12-12T01:58:11.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 547393536 bytes 2023-12-12T01:58:11.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 572065792 bytes 2023-12-12T01:58:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 617352192 bytes 2023-12-12T01:58:11.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 645376000 bytes 2023-12-12T01:58:12.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 725154816 bytes 2023-12-12T01:58:12.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 759145472 bytes 2023-12-12T01:58:12.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 816868352 bytes 2023-12-12T01:58:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 829586432 bytes 2023-12-12T01:58:12.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 861158400 bytes 2023-12-12T01:58:13.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 931697664 bytes 2023-12-12T01:58:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 970361856 bytes 2023-12-12T01:58:13.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1036040192 bytes 2023-12-12T01:58:13.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1102449664 bytes 2023-12-12T01:58:13.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1122344960 bytes 2023-12-12T01:58:14.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1181847552 bytes 2023-12-12T01:58:14.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1194826752 bytes 2023-12-12T01:58:14.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1243761664 bytes 2023-12-12T01:58:14.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1273520128 bytes 2023-12-12T01:58:14.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1298593792 bytes 2023-12-12T01:58:15.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1389676544 bytes 2023-12-12T01:58:15.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1451663360 bytes 2023-12-12T01:58:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1504826368 bytes 2023-12-12T01:58:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1516819456 bytes 2023-12-12T01:58:15.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1560522752 bytes 2023-12-12T01:58:15.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1585191936 bytes 2023-12-12T01:58:16.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1585810432 bytes 2023-12-12T01:58:16.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1622069248 bytes 2023-12-12T01:58:16.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1718199296 bytes 2023-12-12T01:58:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1753831424 bytes 2023-12-12T01:58:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1791973376 bytes 2023-12-12T01:58:17.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1856044032 bytes 2023-12-12T01:58:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1921737728 bytes 2023-12-12T01:58:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1964805120 bytes 2023-12-12T01:58:17.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1969713152 bytes 2023-12-12T01:58:17.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2036517888 bytes 2023-12-12T01:58:18.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2053739520 bytes 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_19 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2023-12-12T01:58:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_19 128M 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_19 2023-12-12T01:58:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:58:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:58:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:18.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2023-12-12T01:58:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T01:58:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:18.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2023-12-12T01:58:19.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:58:19.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2023-12-12T01:58:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_19': 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 132247a37a99 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.132247a37a99 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:18 2023 2023-12-12T01:58:19.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:18 2023 2023-12-12T01:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:18 2023 2023-12-12T01:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 963f10a2-9317-4e5e-b35a-fc335be2a14a 2023-12-12T01:58:19.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:58:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:58:19.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_19 2023-12-12T01:58:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T01:58:19.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:58:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T01:58:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:58:19.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:58:19.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 74847232 bytes 2023-12-12T01:58:19.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 75451392 bytes 2023-12-12T01:58:19.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 104295424 bytes 2023-12-12T01:58:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 150926336 bytes 2023-12-12T01:58:20.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 223064064 bytes 2023-12-12T01:58:20.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 240495616 bytes 2023-12-12T01:58:20.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 262238208 bytes 2023-12-12T01:58:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 317287424 bytes 2023-12-12T01:58:20.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 375788544 bytes 2023-12-12T01:58:20.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 409526272 bytes 2023-12-12T01:58:21.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 458720256 bytes 2023-12-12T01:58:21.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 566170624 bytes 2023-12-12T01:58:21.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 609699840 bytes 2023-12-12T01:58:21.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 610332672 bytes 2023-12-12T01:58:21.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 615431168 bytes 2023-12-12T01:58:22.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 671908864 bytes 2023-12-12T01:58:22.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 778898432 bytes 2023-12-12T01:58:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 848876544 bytes 2023-12-12T01:58:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 915472384 bytes 2023-12-12T01:58:23.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 944002048 bytes 2023-12-12T01:58:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 985999360 bytes 2023-12-12T01:58:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 987648000 bytes 2023-12-12T01:58:23.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1008865280 bytes 2023-12-12T01:58:23.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1061365760 bytes 2023-12-12T01:58:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1143797760 bytes 2023-12-12T01:58:24.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1174929408 bytes 2023-12-12T01:58:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1264354304 bytes 2023-12-12T01:58:24.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1288534016 bytes 2023-12-12T01:58:24.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1434067968 bytes 2023-12-12T01:58:24.983 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T01:58:24.983 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T01:58:24.984 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-12T01:58:24.984 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T01:58:24.988+0000 7fed4f4bd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 47056) UID: 1000 2023-12-12T01:58:25.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1450778624 bytes 2023-12-12T01:58:25.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1451397120 bytes 2023-12-12T01:58:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1506402304 bytes 2023-12-12T01:58:25.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1575173120 bytes 2023-12-12T01:58:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1635426304 bytes 2023-12-12T01:58:25.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1673920512 bytes 2023-12-12T01:58:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1739466752 bytes 2023-12-12T01:58:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1748199424 bytes 2023-12-12T01:58:26.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1770406912 bytes 2023-12-12T01:58:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1831823360 bytes 2023-12-12T01:58:26.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1889738752 bytes 2023-12-12T01:58:27.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1919850496 bytes 2023-12-12T01:58:27.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1973123072 bytes 2023-12-12T01:58:27.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 2008567808 bytes 2023-12-12T01:58:27.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2061753344 bytes 2023-12-12T01:58:27.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2062717952 bytes 2023-12-12T01:58:28.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2121977856 bytes 2023-12-12T01:58:28.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_20 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_20 128M 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:28.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_20 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2023-12-12T01:58:28.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T01:58:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2023-12-12T01:58:29.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:58:29.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2023-12-12T01:58:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_20': 2023-12-12T01:58:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 132a4589d388 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.132a4589d388 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:28 2023 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:28 2023 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:28 2023 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:58:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 74a54c46-73ec-4d02-aaa0-86f3e226c449 2023-12-12T01:58:29.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:58:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:58:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_20 2023-12-12T01:58:29.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T01:58:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:58:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T01:58:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:58:29.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:58:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 92319744 bytes 2023-12-12T01:58:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 116568064 bytes 2023-12-12T01:58:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 150568960 bytes 2023-12-12T01:58:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 214612992 bytes 2023-12-12T01:58:30.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 296782848 bytes 2023-12-12T01:58:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 329044992 bytes 2023-12-12T01:58:30.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 359699456 bytes 2023-12-12T01:58:30.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 405753856 bytes 2023-12-12T01:58:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 455992320 bytes 2023-12-12T01:58:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 532514816 bytes 2023-12-12T01:58:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 580360192 bytes 2023-12-12T01:58:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 660985856 bytes 2023-12-12T01:58:31.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 700048384 bytes 2023-12-12T01:58:32.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 774256640 bytes 2023-12-12T01:58:32.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 774847488 bytes 2023-12-12T01:58:32.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 809976832 bytes 2023-12-12T01:58:32.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 854950912 bytes 2023-12-12T01:58:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 861011968 bytes 2023-12-12T01:58:32.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 918772736 bytes 2023-12-12T01:58:33.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 955577344 bytes 2023-12-12T01:58:33.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1008931840 bytes 2023-12-12T01:58:33.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1061047296 bytes 2023-12-12T01:58:33.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1067325440 bytes 2023-12-12T01:58:33.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1067936768 bytes 2023-12-12T01:58:33.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1128501248 bytes 2023-12-12T01:58:33.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5122491009878865) > weight (0.25) 2023-12-12T01:58:33.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5731333164592611) > weight (0.25) 2023-12-12T01:58:33.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7696079158915661) > weight (0.25) 2023-12-12T01:58:33.969 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6888279540720696) > weight (0.25) 2023-12-12T01:58:33.969 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-12T01:58:33.985 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T01:58:33.985 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T01:58:33.985 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T01:58:33.985 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T01:58:33.986 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T01:58:33.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T01:58:33.989 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-12T01:58:34.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1175814144 bytes 2023-12-12T01:58:34.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1226294272 bytes 2023-12-12T01:58:34.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1270310912 bytes 2023-12-12T01:58:34.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1278133248 bytes 2023-12-12T01:58:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1291952128 bytes 2023-12-12T01:58:34.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1313207296 bytes 2023-12-12T01:58:35.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1335331840 bytes 2023-12-12T01:58:35.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1409165312 bytes 2023-12-12T01:58:35.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1464024064 bytes 2023-12-12T01:58:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1489684480 bytes 2023-12-12T01:58:35.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1564555264 bytes 2023-12-12T01:58:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1619117056 bytes 2023-12-12T01:58:36.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1624195072 bytes 2023-12-12T01:58:36.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1715187712 bytes 2023-12-12T01:58:36.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1725639680 bytes 2023-12-12T01:58:36.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1789532160 bytes 2023-12-12T01:58:37.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1827088384 bytes 2023-12-12T01:58:37.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1868362752 bytes 2023-12-12T01:58:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1915155456 bytes 2023-12-12T01:58:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1941216256 bytes 2023-12-12T01:58:37.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1960160256 bytes 2023-12-12T01:58:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2017150976 bytes 2023-12-12T01:58:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2024406016 bytes 2023-12-12T01:58:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2033546240 bytes 2023-12-12T01:58:38.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2112110592 bytes 2023-12-12T01:58:38.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2175723520 bytes 2023-12-12T01:58:38.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2193008640 bytes 2023-12-12T01:58:39.045 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2218468352 bytes 2023-12-12T01:58:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2219077632 bytes 2023-12-12T01:58:39.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2275410944 bytes 2023-12-12T01:58:39.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2311861248 bytes 2023-12-12T01:58:39.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2379666432 bytes 2023-12-12T01:58:39.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2425527296 bytes 2023-12-12T01:58:40.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_21 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_21 128M 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_21 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:40.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2023-12-12T01:58:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:58:41.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_21': 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 133b68ad2a49 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.133b68ad2a49 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:58:41.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:40 2023 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:40 2023 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:40 2023 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 59ede7a8-6346-4103-a0a4-f50800c189ac 2023-12-12T01:58:41.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_21 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T01:58:41.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:58:41.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:58:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:58:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:58:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 26278912 bytes 2023-12-12T01:58:41.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 33905664 bytes 2023-12-12T01:58:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 54172672 bytes 2023-12-12T01:58:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 72747008 bytes 2023-12-12T01:58:41.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 132280320 bytes 2023-12-12T01:58:42.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 132914176 bytes 2023-12-12T01:58:42.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 247109632 bytes 2023-12-12T01:58:42.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 256520192 bytes 2023-12-12T01:58:42.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 287781888 bytes 2023-12-12T01:58:42.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 336128000 bytes 2023-12-12T01:58:42.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 336748544 bytes 2023-12-12T01:58:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 366667776 bytes 2023-12-12T01:58:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 398955520 bytes 2023-12-12T01:58:43.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 483036160 bytes 2023-12-12T01:58:43.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 507925504 bytes 2023-12-12T01:58:43.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 553146368 bytes 2023-12-12T01:58:43.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 560738304 bytes 2023-12-12T01:58:44.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 638332928 bytes 2023-12-12T01:58:44.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 680136704 bytes 2023-12-12T01:58:44.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 731566080 bytes 2023-12-12T01:58:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 774746112 bytes 2023-12-12T01:58:44.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 811515904 bytes 2023-12-12T01:58:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 858241024 bytes 2023-12-12T01:58:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 886792192 bytes 2023-12-12T01:58:45.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 902978560 bytes 2023-12-12T01:58:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 963810304 bytes 2023-12-12T01:58:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 990619648 bytes 2023-12-12T01:58:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1035515904 bytes 2023-12-12T01:58:46.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1057863680 bytes 2023-12-12T01:58:46.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1066062848 bytes 2023-12-12T01:58:46.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1116008448 bytes 2023-12-12T01:58:46.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1196282880 bytes 2023-12-12T01:58:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1221410816 bytes 2023-12-12T01:58:47.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1234757632 bytes 2023-12-12T01:58:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1246686208 bytes 2023-12-12T01:58:47.218 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T01:58:47.219+0000 7fd7ab621640 -1 reset not still connected to 0x557463ab5860 2023-12-12T01:58:47.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1298737152 bytes 2023-12-12T01:58:47.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1322985472 bytes 2023-12-12T01:58:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1386524672 bytes 2023-12-12T01:58:47.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1461138432 bytes 2023-12-12T01:58:48.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1521349632 bytes 2023-12-12T01:58:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1542845440 bytes 2023-12-12T01:58:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1543450624 bytes 2023-12-12T01:58:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1563750400 bytes 2023-12-12T01:58:48.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1598055424 bytes 2023-12-12T01:58:48.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1646689280 bytes 2023-12-12T01:58:49.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1696752640 bytes 2023-12-12T01:58:49.407 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1740510208 bytes 2023-12-12T01:58:49.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1809792000 bytes 2023-12-12T01:58:49.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1820469248 bytes 2023-12-12T01:58:49.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1853638656 bytes 2023-12-12T01:58:50.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1855486976 bytes 2023-12-12T01:58:50.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 1934715904 bytes 2023-12-12T01:58:50.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 1953651712 bytes 2023-12-12T01:58:50.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 1997244416 bytes 2023-12-12T01:58:50.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2024573952 bytes 2023-12-12T01:58:51.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_22 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_22 128M 2023-12-12T01:58:51.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:58:51.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_22 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:58:51.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2023-12-12T01:58:52.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:58:52.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_22': 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1345d625420e 2023-12-12T01:58:52.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1345d625420e 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:51 2023 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:51 2023 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:51 2023 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: e51faaaf-277b-490b-9e12-ad199fdee9c2 2023-12-12T01:58:52.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:58:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:58:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_22 2023-12-12T01:58:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:58:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:58:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T01:58:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:58:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T01:58:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:58:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:58:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 71641088 bytes 2023-12-12T01:58:52.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 90047488 bytes 2023-12-12T01:58:52.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 155953152 bytes 2023-12-12T01:58:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 159568896 bytes 2023-12-12T01:58:53.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 215001088 bytes 2023-12-12T01:58:53.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 251877376 bytes 2023-12-12T01:58:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 291663872 bytes 2023-12-12T01:58:53.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 338321408 bytes 2023-12-12T01:58:54.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 357700608 bytes 2023-12-12T01:58:54.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 411768832 bytes 2023-12-12T01:58:54.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 413516800 bytes 2023-12-12T01:58:54.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 451709952 bytes 2023-12-12T01:58:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 468333568 bytes 2023-12-12T01:58:54.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 471721984 bytes 2023-12-12T01:58:55.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 517817344 bytes 2023-12-12T01:58:55.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 527864832 bytes 2023-12-12T01:58:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 562807808 bytes 2023-12-12T01:58:55.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 578148352 bytes 2023-12-12T01:58:55.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 592722944 bytes 2023-12-12T01:58:55.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 606451712 bytes 2023-12-12T01:58:55.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 623457280 bytes 2023-12-12T01:58:56.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 670228480 bytes 2023-12-12T01:58:56.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 691761152 bytes 2023-12-12T01:58:56.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 715335680 bytes 2023-12-12T01:58:56.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 736269312 bytes 2023-12-12T01:58:56.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 744912896 bytes 2023-12-12T01:58:56.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 788108288 bytes 2023-12-12T01:58:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 819516416 bytes 2023-12-12T01:58:57.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 871929856 bytes 2023-12-12T01:58:57.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 945035264 bytes 2023-12-12T01:58:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 979230720 bytes 2023-12-12T01:58:57.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 991157248 bytes 2023-12-12T01:58:57.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 997752832 bytes 2023-12-12T01:58:58.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1057399808 bytes 2023-12-12T01:58:58.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1090697216 bytes 2023-12-12T01:58:58.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1095398400 bytes 2023-12-12T01:58:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1173250048 bytes 2023-12-12T01:58:58.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1186872320 bytes 2023-12-12T01:58:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1222774784 bytes 2023-12-12T01:58:59.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1269146624 bytes 2023-12-12T01:58:59.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1295443968 bytes 2023-12-12T01:58:59.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1328035840 bytes 2023-12-12T01:58:59.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1352561664 bytes 2023-12-12T01:58:59.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1408471040 bytes 2023-12-12T01:58:59.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1433431040 bytes 2023-12-12T01:59:00.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1507205120 bytes 2023-12-12T01:59:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:00.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_23 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_23 128M 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:00.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_23 2023-12-12T01:59:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:00.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:00.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2023-12-12T01:59:01.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:01.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2023-12-12T01:59:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_23': 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 134de82a21ad 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.134de82a21ad 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:00 2023 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:00 2023 2023-12-12T01:59:01.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:00 2023 2023-12-12T01:59:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a4a6e595-cc0c-4cac-b030-d274e817dada 2023-12-12T01:59:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_23 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T01:59:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T01:59:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:01.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:02.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 50699264 bytes 2023-12-12T01:59:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 101379072 bytes 2023-12-12T01:59:02.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 115119104 bytes 2023-12-12T01:59:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 185869312 bytes 2023-12-12T01:59:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 235684864 bytes 2023-12-12T01:59:02.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 241351680 bytes 2023-12-12T01:59:03.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 266940416 bytes 2023-12-12T01:59:03.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 267537408 bytes 2023-12-12T01:59:03.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 320885760 bytes 2023-12-12T01:59:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 321486848 bytes 2023-12-12T01:59:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 346916864 bytes 2023-12-12T01:59:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 408646656 bytes 2023-12-12T01:59:04.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 469124096 bytes 2023-12-12T01:59:04.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 489653248 bytes 2023-12-12T01:59:04.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 521885696 bytes 2023-12-12T01:59:04.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 540462080 bytes 2023-12-12T01:59:04.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 594681856 bytes 2023-12-12T01:59:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 644260864 bytes 2023-12-12T01:59:05.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 706828288 bytes 2023-12-12T01:59:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 744458240 bytes 2023-12-12T01:59:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 798732288 bytes 2023-12-12T01:59:05.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 829417472 bytes 2023-12-12T01:59:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 853210112 bytes 2023-12-12T01:59:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 890997760 bytes 2023-12-12T01:59:06.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 891607040 bytes 2023-12-12T01:59:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 927222784 bytes 2023-12-12T01:59:06.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1000752128 bytes 2023-12-12T01:59:06.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1018171392 bytes 2023-12-12T01:59:07.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1079189504 bytes 2023-12-12T01:59:07.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1106241536 bytes 2023-12-12T01:59:07.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1142333440 bytes 2023-12-12T01:59:07.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1162483712 bytes 2023-12-12T01:59:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1172027392 bytes 2023-12-12T01:59:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1221509120 bytes 2023-12-12T01:59:08.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1285984256 bytes 2023-12-12T01:59:08.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1366671360 bytes 2023-12-12T01:59:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1395086336 bytes 2023-12-12T01:59:08.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1423887360 bytes 2023-12-12T01:59:08.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1443184640 bytes 2023-12-12T01:59:09.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1488543744 bytes 2023-12-12T01:59:09.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1554231296 bytes 2023-12-12T01:59:09.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1595958272 bytes 2023-12-12T01:59:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:09.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_24 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_24 128M 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:09.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_24 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2023-12-12T01:59:09.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T01:59:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2023-12-12T01:59:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_24': 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 135998871cd0 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.135998871cd0 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:09 2023 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:09 2023 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:09 2023 2023-12-12T01:59:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d213a902-dc35-4507-886a-3f4b1129d6d2 2023-12-12T01:59:10.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_24 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T01:59:10.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T01:59:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:10.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:10.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 10763264 bytes 2023-12-12T01:59:10.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 35843072 bytes 2023-12-12T01:59:10.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 49093632 bytes 2023-12-12T01:59:10.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 80475136 bytes 2023-12-12T01:59:11.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 156974080 bytes 2023-12-12T01:59:11.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 157586432 bytes 2023-12-12T01:59:11.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 158185472 bytes 2023-12-12T01:59:11.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 175875072 bytes 2023-12-12T01:59:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 187810816 bytes 2023-12-12T01:59:11.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 218884096 bytes 2023-12-12T01:59:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 252898304 bytes 2023-12-12T01:59:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 317902848 bytes 2023-12-12T01:59:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 380092416 bytes 2023-12-12T01:59:12.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 405889024 bytes 2023-12-12T01:59:12.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 483441664 bytes 2023-12-12T01:59:13.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 590171136 bytes 2023-12-12T01:59:13.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 630800384 bytes 2023-12-12T01:59:13.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 690677760 bytes 2023-12-12T01:59:13.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 748793856 bytes 2023-12-12T01:59:14.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 842675200 bytes 2023-12-12T01:59:14.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 924100608 bytes 2023-12-12T01:59:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 992652288 bytes 2023-12-12T01:59:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1058151424 bytes 2023-12-12T01:59:15.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1061184512 bytes 2023-12-12T01:59:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1067783168 bytes 2023-12-12T01:59:15.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1203616768 bytes 2023-12-12T01:59:15.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1240550400 bytes 2023-12-12T01:59:15.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1263294464 bytes 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_25 2023-12-12T01:59:16.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_25 128M 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:16.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_25 2023-12-12T01:59:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:16.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:16.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2023-12-12T01:59:16.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_25': 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 136269b07e90 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.136269b07e90 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:16 2023 2023-12-12T01:59:16.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:16 2023 2023-12-12T01:59:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:16 2023 2023-12-12T01:59:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a921f597-6426-4126-a233-e3c16f3533b3 2023-12-12T01:59:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_25 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T01:59:16.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T01:59:16.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:16.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:16.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 37047296 bytes 2023-12-12T01:59:17.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 108866560 bytes 2023-12-12T01:59:17.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 135431168 bytes 2023-12-12T01:59:17.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 228801536 bytes 2023-12-12T01:59:17.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 283865088 bytes 2023-12-12T01:59:17.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 305697792 bytes 2023-12-12T01:59:17.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 329559040 bytes 2023-12-12T01:59:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 390679552 bytes 2023-12-12T01:59:18.407 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 455454720 bytes 2023-12-12T01:59:18.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 506370048 bytes 2023-12-12T01:59:18.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 547294208 bytes 2023-12-12T01:59:19.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 616681472 bytes 2023-12-12T01:59:19.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 652606464 bytes 2023-12-12T01:59:19.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 714050560 bytes 2023-12-12T01:59:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 716061696 bytes 2023-12-12T01:59:19.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 780737536 bytes 2023-12-12T01:59:19.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 810516480 bytes 2023-12-12T01:59:20.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 834779136 bytes 2023-12-12T01:59:20.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 880239616 bytes 2023-12-12T01:59:20.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 881891328 bytes 2023-12-12T01:59:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 941221888 bytes 2023-12-12T01:59:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 995544064 bytes 2023-12-12T01:59:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1032962048 bytes 2023-12-12T01:59:21.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1042871296 bytes 2023-12-12T01:59:21.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1097116672 bytes 2023-12-12T01:59:21.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1127794688 bytes 2023-12-12T01:59:21.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1143279616 bytes 2023-12-12T01:59:21.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1212572672 bytes 2023-12-12T01:59:22.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1237008384 bytes 2023-12-12T01:59:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1312663552 bytes 2023-12-12T01:59:22.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1367214080 bytes 2023-12-12T01:59:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1420098560 bytes 2023-12-12T01:59:23.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1484984320 bytes 2023-12-12T01:59:23.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1511074816 bytes 2023-12-12T01:59:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1575278592 bytes 2023-12-12T01:59:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1608109056 bytes 2023-12-12T01:59:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1623947264 bytes 2023-12-12T01:59:23.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1662574592 bytes 2023-12-12T01:59:24.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1723762688 bytes 2023-12-12T01:59:24.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1733392384 bytes 2023-12-12T01:59:24.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1859786752 bytes 2023-12-12T01:59:24.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1946802176 bytes 2023-12-12T01:59:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1973075968 bytes 2023-12-12T01:59:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1990569984 bytes 2023-12-12T01:59:25.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2030019584 bytes 2023-12-12T01:59:25.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2072706048 bytes 2023-12-12T01:59:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2120813568 bytes 2023-12-12T01:59:25.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2176102400 bytes 2023-12-12T01:59:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2211201024 bytes 2023-12-12T01:59:26.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2239844352 bytes 2023-12-12T01:59:26.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2283716608 bytes 2023-12-12T01:59:26.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2301757440 bytes 2023-12-12T01:59:26.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2374240256 bytes 2023-12-12T01:59:27.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2454005760 bytes 2023-12-12T01:59:27.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2505820160 bytes 2023-12-12T01:59:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2545454080 bytes 2023-12-12T01:59:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2598931456 bytes 2023-12-12T01:59:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2662815744 bytes 2023-12-12T01:59:28.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2706935808 bytes 2023-12-12T01:59:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2761399296 bytes 2023-12-12T01:59:28.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2826797056 bytes 2023-12-12T01:59:28.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2864063488 bytes 2023-12-12T01:59:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2864672768 bytes 2023-12-12T01:59:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2925427712 bytes 2023-12-12T01:59:29.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2940849152 bytes 2023-12-12T01:59:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2961712128 bytes 2023-12-12T01:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:29.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_26 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_26 128M 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:29.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:29.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_26 2023-12-12T01:59:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:29.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2023-12-12T01:59:30.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2023-12-12T01:59:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_26': 2023-12-12T01:59:30.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 136c22c30ad5 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.136c22c30ad5 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:29 2023 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:29 2023 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:29 2023 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 26f2fa83-50d9-4f27-b56d-3181fdc135a5 2023-12-12T01:59:30.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:30.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_26 2023-12-12T01:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T01:59:30.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:30.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=12 2023-12-12T01:59:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:30.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 31419392 bytes 2023-12-12T01:59:30.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 37494784 bytes 2023-12-12T01:59:30.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 57027584 bytes 2023-12-12T01:59:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 92548096 bytes 2023-12-12T01:59:30.840 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 107299840 bytes 2023-12-12T01:59:31.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 200684544 bytes 2023-12-12T01:59:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 209235968 bytes 2023-12-12T01:59:31.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 210369536 bytes 2023-12-12T01:59:31.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 234809344 bytes 2023-12-12T01:59:31.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 256842752 bytes 2023-12-12T01:59:31.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 303206400 bytes 2023-12-12T01:59:31.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 328850432 bytes 2023-12-12T01:59:32.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 362166272 bytes 2023-12-12T01:59:32.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 447591424 bytes 2023-12-12T01:59:32.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 473701376 bytes 2023-12-12T01:59:32.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 520224768 bytes 2023-12-12T01:59:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 539793408 bytes 2023-12-12T01:59:33.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 604157952 bytes 2023-12-12T01:59:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 653236224 bytes 2023-12-12T01:59:33.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 687668224 bytes 2023-12-12T01:59:33.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 727556096 bytes 2023-12-12T01:59:33.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 748665856 bytes 2023-12-12T01:59:34.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 778544128 bytes 2023-12-12T01:59:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 902005760 bytes 2023-12-12T01:59:34.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 924459008 bytes 2023-12-12T01:59:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 979264512 bytes 2023-12-12T01:59:34.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1037302784 bytes 2023-12-12T01:59:35.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1083275264 bytes 2023-12-12T01:59:35.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1149247488 bytes 2023-12-12T01:59:35.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1197865984 bytes 2023-12-12T01:59:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1211066368 bytes 2023-12-12T01:59:35.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1253933056 bytes 2023-12-12T01:59:36.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1281878016 bytes 2023-12-12T01:59:36.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1301652480 bytes 2023-12-12T01:59:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1331998720 bytes 2023-12-12T01:59:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1412304896 bytes 2023-12-12T01:59:36.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1452585984 bytes 2023-12-12T01:59:37.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1483532288 bytes 2023-12-12T01:59:37.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1521323008 bytes 2023-12-12T01:59:37.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1558659072 bytes 2023-12-12T01:59:37.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1559279616 bytes 2023-12-12T01:59:37.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1590683648 bytes 2023-12-12T01:59:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1642813440 bytes 2023-12-12T01:59:38.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1696338944 bytes 2023-12-12T01:59:38.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1724701696 bytes 2023-12-12T01:59:38.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1791964160 bytes 2023-12-12T01:59:38.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1871664128 bytes 2023-12-12T01:59:38.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1905115136 bytes 2023-12-12T01:59:39.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1998810112 bytes 2023-12-12T01:59:39.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2030638080 bytes 2023-12-12T01:59:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2065989632 bytes 2023-12-12T01:59:39.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2109225984 bytes 2023-12-12T01:59:39.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2187681792 bytes 2023-12-12T01:59:40.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2205786112 bytes 2023-12-12T01:59:40.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2253757440 bytes 2023-12-12T01:59:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2296205312 bytes 2023-12-12T01:59:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2296825856 bytes 2023-12-12T01:59:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2363742208 bytes 2023-12-12T01:59:41.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2380451840 bytes 2023-12-12T01:59:41.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2427747328 bytes 2023-12-12T01:59:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2479241216 bytes 2023-12-12T01:59:41.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2520621056 bytes 2023-12-12T01:59:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2530455552 bytes 2023-12-12T01:59:41.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2583330816 bytes 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_27 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:42.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_27 128M 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:42.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:42.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_27 2023-12-12T01:59:42.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:42.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2023-12-12T01:59:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:43.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_27': 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1375865fd2c8 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1375865fd2c8 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:42 2023 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:42 2023 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:42 2023 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 82b98cb4-5a2a-4924-8f4e-9681fc9a7a67 2023-12-12T01:59:43.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_27 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T01:59:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T01:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:43.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:43.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 59931648 bytes 2023-12-12T01:59:43.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 126551040 bytes 2023-12-12T01:59:43.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 188332032 bytes 2023-12-12T01:59:43.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 197808128 bytes 2023-12-12T01:59:44.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 279757824 bytes 2023-12-12T01:59:44.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 306649088 bytes 2023-12-12T01:59:44.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 315388928 bytes 2023-12-12T01:59:44.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 363015168 bytes 2023-12-12T01:59:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 402768896 bytes 2023-12-12T01:59:45.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 413583360 bytes 2023-12-12T01:59:45.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 485725184 bytes 2023-12-12T01:59:45.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 509251584 bytes 2023-12-12T01:59:45.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 533458944 bytes 2023-12-12T01:59:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 557980672 bytes 2023-12-12T01:59:45.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 591079424 bytes 2023-12-12T01:59:46.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 655019008 bytes 2023-12-12T01:59:46.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 716210176 bytes 2023-12-12T01:59:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 754257920 bytes 2023-12-12T01:59:46.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 778184704 bytes 2023-12-12T01:59:46.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 833992704 bytes 2023-12-12T01:59:47.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 885978112 bytes 2023-12-12T01:59:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 924516352 bytes 2023-12-12T01:59:47.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 980049920 bytes 2023-12-12T01:59:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1031898112 bytes 2023-12-12T01:59:47.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1068289024 bytes 2023-12-12T01:59:48.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1099506688 bytes 2023-12-12T01:59:48.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1149402112 bytes 2023-12-12T01:59:48.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1190392832 bytes 2023-12-12T01:59:48.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1220343808 bytes 2023-12-12T01:59:48.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1287883776 bytes 2023-12-12T01:59:48.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5897465390137476) > weight (0.25) 2023-12-12T01:59:48.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.47374506803668914) > weight (0.25) 2023-12-12T01:59:48.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7731744962966844) > weight (0.25) 2023-12-12T01:59:48.990 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3235639738154249) > weight (0.25) 2023-12-12T01:59:48.990 INFO:tasks.rbd_mirror_thrash:waiting for 110 secs before thrashing 2023-12-12T01:59:49.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1296399360 bytes 2023-12-12T01:59:49.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1347284992 bytes 2023-12-12T01:59:49.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1399718912 bytes 2023-12-12T01:59:49.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1449508864 bytes 2023-12-12T01:59:49.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1490668544 bytes 2023-12-12T01:59:50.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1518110720 bytes 2023-12-12T01:59:50.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1533136896 bytes 2023-12-12T01:59:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1557520384 bytes 2023-12-12T01:59:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1562527744 bytes 2023-12-12T01:59:50.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1595615232 bytes 2023-12-12T01:59:50.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1621153792 bytes 2023-12-12T01:59:51.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1718962176 bytes 2023-12-12T01:59:51.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1792956416 bytes 2023-12-12T01:59:51.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1793563648 bytes 2023-12-12T01:59:51.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1830755328 bytes 2023-12-12T01:59:51.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1903801344 bytes 2023-12-12T01:59:52.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1928776704 bytes 2023-12-12T01:59:52.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2002057216 bytes 2023-12-12T01:59:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2032456704 bytes 2023-12-12T01:59:52.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2055666688 bytes 2023-12-12T01:59:52.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2129488896 bytes 2023-12-12T01:59:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2156477440 bytes 2023-12-12T01:59:53.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2194428928 bytes 2023-12-12T01:59:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2233996288 bytes 2023-12-12T01:59:53.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2288901120 bytes 2023-12-12T01:59:53.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2312785920 bytes 2023-12-12T01:59:54.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2357391360 bytes 2023-12-12T01:59:54.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_28 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T01:59:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_28 128M 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T01:59:54.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_28 2023-12-12T01:59:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T01:59:54.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2023-12-12T01:59:55.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T01:59:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2023-12-12T01:59:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_28': 2023-12-12T01:59:55.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 137e8015a99c 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.137e8015a99c 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:54 2023 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:54 2023 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:54 2023 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c9a7b9aa-ebf2-4ce9-8f2a-36162c41d7cf 2023-12-12T01:59:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T01:59:55.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_28 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T01:59:55.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2023-12-12T01:59:55.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 21736448 bytes 2023-12-12T01:59:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 66037760 bytes 2023-12-12T01:59:55.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 119506944 bytes 2023-12-12T01:59:55.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 162873344 bytes 2023-12-12T01:59:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 170055680 bytes 2023-12-12T01:59:56.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 201718784 bytes 2023-12-12T01:59:56.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 249999360 bytes 2023-12-12T01:59:56.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 344616960 bytes 2023-12-12T01:59:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 362977280 bytes 2023-12-12T01:59:57.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 436960256 bytes 2023-12-12T01:59:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 480039936 bytes 2023-12-12T01:59:57.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 585054208 bytes 2023-12-12T01:59:57.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 629511168 bytes 2023-12-12T01:59:57.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 664466432 bytes 2023-12-12T01:59:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 701379584 bytes 2023-12-12T01:59:58.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 727043072 bytes 2023-12-12T01:59:58.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 757463040 bytes 2023-12-12T01:59:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 821243904 bytes 2023-12-12T01:59:58.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 897747968 bytes 2023-12-12T01:59:59.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 950793216 bytes 2023-12-12T01:59:59.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 971783168 bytes 2023-12-12T01:59:59.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1002830848 bytes 2023-12-12T01:59:59.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1022053376 bytes 2023-12-12T01:59:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1078778880 bytes 2023-12-12T01:59:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1114029056 bytes 2023-12-12T02:00:00.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1169693696 bytes 2023-12-12T02:00:00.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1184881664 bytes 2023-12-12T02:00:00.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1217323008 bytes 2023-12-12T02:00:00.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1290843136 bytes 2023-12-12T02:00:00.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1347035136 bytes 2023-12-12T02:00:01.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1347630080 bytes 2023-12-12T02:00:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1385529344 bytes 2023-12-12T02:00:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1386117120 bytes 2023-12-12T02:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1389103104 bytes 2023-12-12T02:00:01.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1451535360 bytes 2023-12-12T02:00:01.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1491202048 bytes 2023-12-12T02:00:02.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1509283840 bytes 2023-12-12T02:00:02.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1547894784 bytes 2023-12-12T02:00:02.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1575718912 bytes 2023-12-12T02:00:02.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1637300224 bytes 2023-12-12T02:00:02.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1637919744 bytes 2023-12-12T02:00:02.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1665153024 bytes 2023-12-12T02:00:03.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1748664320 bytes 2023-12-12T02:00:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1772768256 bytes 2023-12-12T02:00:03.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1794370560 bytes 2023-12-12T02:00:03.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1808011264 bytes 2023-12-12T02:00:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1843120128 bytes 2023-12-12T02:00:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1870384128 bytes 2023-12-12T02:00:04.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1908455424 bytes 2023-12-12T02:00:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1925229568 bytes 2023-12-12T02:00:04.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1957924864 bytes 2023-12-12T02:00:04.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2000340992 bytes 2023-12-12T02:00:04.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2054150144 bytes 2023-12-12T02:00:05.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2149989376 bytes 2023-12-12T02:00:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:05.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_29 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_29 128M 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:05.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:05.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_29 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:05.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2023-12-12T02:00:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:06.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2023-12-12T02:00:06.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_29': 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1387e4e4de4e 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1387e4e4de4e 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:05 2023 2023-12-12T02:00:06.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:05 2023 2023-12-12T02:00:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:05 2023 2023-12-12T02:00:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d3da7272-827d-4801-89f4-6e6b377cdca7 2023-12-12T02:00:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_29 2023-12-12T02:00:06.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:00:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=12 2023-12-12T02:00:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:06.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 611328 bytes 2023-12-12T02:00:06.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 27475968 bytes 2023-12-12T02:00:06.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 46518272 bytes 2023-12-12T02:00:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 70616064 bytes 2023-12-12T02:00:06.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 104123392 bytes 2023-12-12T02:00:07.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 104725504 bytes 2023-12-12T02:00:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 124181504 bytes 2023-12-12T02:00:07.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 144031744 bytes 2023-12-12T02:00:07.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 222998528 bytes 2023-12-12T02:00:07.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 282275840 bytes 2023-12-12T02:00:08.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 414621696 bytes 2023-12-12T02:00:08.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 459283456 bytes 2023-12-12T02:00:08.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 464520192 bytes 2023-12-12T02:00:08.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 500504576 bytes 2023-12-12T02:00:08.935 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 552640512 bytes 2023-12-12T02:00:09.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 604413952 bytes 2023-12-12T02:00:09.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 654587904 bytes 2023-12-12T02:00:09.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 684754944 bytes 2023-12-12T02:00:09.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 709107712 bytes 2023-12-12T02:00:09.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 746458112 bytes 2023-12-12T02:00:09.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 770001920 bytes 2023-12-12T02:00:10.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 838294528 bytes 2023-12-12T02:00:10.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 853446656 bytes 2023-12-12T02:00:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 941222912 bytes 2023-12-12T02:00:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 983279616 bytes 2023-12-12T02:00:11.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1030146048 bytes 2023-12-12T02:00:11.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1058872320 bytes 2023-12-12T02:00:11.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1091758080 bytes 2023-12-12T02:00:11.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1097904128 bytes 2023-12-12T02:00:11.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1101815808 bytes 2023-12-12T02:00:11.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1137982464 bytes 2023-12-12T02:00:11.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1187147776 bytes 2023-12-12T02:00:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1197390848 bytes 2023-12-12T02:00:12.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1233284096 bytes 2023-12-12T02:00:12.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1327352832 bytes 2023-12-12T02:00:12.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1362809856 bytes 2023-12-12T02:00:12.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1408529408 bytes 2023-12-12T02:00:12.974 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9828672969422426) > weight (0.25) 2023-12-12T02:00:12.974 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T02:00:12.974 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T02:00:12.975 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T02:00:12.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:00:12.982+0000 7f0045316640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43668) UID: 1000 2023-12-12T02:00:13.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1487448064 bytes 2023-12-12T02:00:13.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1490929664 bytes 2023-12-12T02:00:13.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1535805440 bytes 2023-12-12T02:00:13.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1587369984 bytes 2023-12-12T02:00:13.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1605447680 bytes 2023-12-12T02:00:14.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1632266240 bytes 2023-12-12T02:00:14.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1635749888 bytes 2023-12-12T02:00:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1685248000 bytes 2023-12-12T02:00:14.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1745381376 bytes 2023-12-12T02:00:14.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1781148672 bytes 2023-12-12T02:00:15.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1792417792 bytes 2023-12-12T02:00:15.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1849402368 bytes 2023-12-12T02:00:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1866925056 bytes 2023-12-12T02:00:15.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1885237248 bytes 2023-12-12T02:00:15.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 1943355392 bytes 2023-12-12T02:00:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2002974720 bytes 2023-12-12T02:00:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2028041216 bytes 2023-12-12T02:00:16.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2085555200 bytes 2023-12-12T02:00:16.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2098784256 bytes 2023-12-12T02:00:16.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2099403776 bytes 2023-12-12T02:00:16.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2128531456 bytes 2023-12-12T02:00:16.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2164063232 bytes 2023-12-12T02:00:16.975 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T02:00:16.976 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:00:16.976 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T02:00:16.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T02:00:16.976 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T02:00:16.979 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T02:00:16.979 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-12T02:00:17.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2233994240 bytes 2023-12-12T02:00:17.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2289120256 bytes 2023-12-12T02:00:17.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2371956736 bytes 2023-12-12T02:00:18.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2455187456 bytes 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_30 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:18.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_30 128M 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:00:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:18.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_30 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:00:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2023-12-12T02:00:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:18.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_30': 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1396abd487a8 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1396abd487a8 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:18 2023 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:18 2023 2023-12-12T02:00:18.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:18 2023 2023-12-12T02:00:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 37a61d89-b375-4db8-b09d-d9947b095e5c 2023-12-12T02:00:18.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:18.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_30 2023-12-12T02:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:00:18.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T02:00:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:18.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 77350912 bytes 2023-12-12T02:00:19.130 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T02:00:19.131+0000 7f2a08a08640 -1 reset not still connected to 0x55c66c1de270 2023-12-12T02:00:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 123282432 bytes 2023-12-12T02:00:19.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 151159808 bytes 2023-12-12T02:00:19.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 191544320 bytes 2023-12-12T02:00:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 270469120 bytes 2023-12-12T02:00:19.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 285469696 bytes 2023-12-12T02:00:20.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 359238656 bytes 2023-12-12T02:00:20.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 372408320 bytes 2023-12-12T02:00:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 492923904 bytes 2023-12-12T02:00:20.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 524295168 bytes 2023-12-12T02:00:20.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 552203264 bytes 2023-12-12T02:00:21.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 577734656 bytes 2023-12-12T02:00:21.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 621865984 bytes 2023-12-12T02:00:21.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 690921472 bytes 2023-12-12T02:00:21.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 784631808 bytes 2023-12-12T02:00:22.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 815234048 bytes 2023-12-12T02:00:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 876750848 bytes 2023-12-12T02:00:22.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 905239552 bytes 2023-12-12T02:00:22.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 919342080 bytes 2023-12-12T02:00:22.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 984067072 bytes 2023-12-12T02:00:23.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1008726016 bytes 2023-12-12T02:00:23.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1048796160 bytes 2023-12-12T02:00:23.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1090247680 bytes 2023-12-12T02:00:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1114582016 bytes 2023-12-12T02:00:23.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1149690880 bytes 2023-12-12T02:00:23.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1177524224 bytes 2023-12-12T02:00:24.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1245138944 bytes 2023-12-12T02:00:24.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1304130560 bytes 2023-12-12T02:00:24.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1326788608 bytes 2023-12-12T02:00:24.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1381841920 bytes 2023-12-12T02:00:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1414432768 bytes 2023-12-12T02:00:25.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1462610944 bytes 2023-12-12T02:00:25.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1508692992 bytes 2023-12-12T02:00:25.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1552724992 bytes 2023-12-12T02:00:25.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1585282048 bytes 2023-12-12T02:00:26.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1621361664 bytes 2023-12-12T02:00:26.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1638787072 bytes 2023-12-12T02:00:26.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1695695872 bytes 2023-12-12T02:00:26.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1723569152 bytes 2023-12-12T02:00:26.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1782838272 bytes 2023-12-12T02:00:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1798440960 bytes 2023-12-12T02:00:27.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1860745216 bytes 2023-12-12T02:00:27.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1909597184 bytes 2023-12-12T02:00:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1914414080 bytes 2023-12-12T02:00:27.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1999918080 bytes 2023-12-12T02:00:28.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2082201600 bytes 2023-12-12T02:00:28.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2153210880 bytes 2023-12-12T02:00:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2225713152 bytes 2023-12-12T02:00:28.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2257372160 bytes 2023-12-12T02:00:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2308276224 bytes 2023-12-12T02:00:29.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2339632128 bytes 2023-12-12T02:00:29.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2384191488 bytes 2023-12-12T02:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:29.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_31 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:29.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_31 128M 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_31 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:29.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2023-12-12T02:00:30.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:30.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2023-12-12T02:00:30.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_31': 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13a08025385c 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13a08025385c 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:29 2023 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:29 2023 2023-12-12T02:00:30.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:29 2023 2023-12-12T02:00:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: f5bbdd5f-8443-4677-b3b4-d498845c7bef 2023-12-12T02:00:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_31 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:00:30.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T02:00:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:30.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:30.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 12248064 bytes 2023-12-12T02:00:30.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 12853248 bytes 2023-12-12T02:00:30.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 55513088 bytes 2023-12-12T02:00:30.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 103035904 bytes 2023-12-12T02:00:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 146083840 bytes 2023-12-12T02:00:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 217393152 bytes 2023-12-12T02:00:31.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 218020864 bytes 2023-12-12T02:00:31.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 251942912 bytes 2023-12-12T02:00:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 309299200 bytes 2023-12-12T02:00:31.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 322452480 bytes 2023-12-12T02:00:31.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 323061760 bytes 2023-12-12T02:00:32.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 411713536 bytes 2023-12-12T02:00:32.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 441268224 bytes 2023-12-12T02:00:32.506 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T02:00:32.513+0000 7f76736a6640 -1 reset not still connected to 0x55a800c28f70 2023-12-12T02:00:32.506 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T02:00:32.513+0000 7f76736a6640 -1 reset not still connected to 0x55a800fd0410 2023-12-12T02:00:32.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 496652288 bytes 2023-12-12T02:00:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 518653952 bytes 2023-12-12T02:00:33.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 599626752 bytes 2023-12-12T02:00:33.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 603782144 bytes 2023-12-12T02:00:33.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 660337664 bytes 2023-12-12T02:00:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 680288256 bytes 2023-12-12T02:00:33.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 760412160 bytes 2023-12-12T02:00:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 781189120 bytes 2023-12-12T02:00:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 815467520 bytes 2023-12-12T02:00:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 832689152 bytes 2023-12-12T02:00:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 879290368 bytes 2023-12-12T02:00:34.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 920742912 bytes 2023-12-12T02:00:34.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 950928384 bytes 2023-12-12T02:00:34.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 957464576 bytes 2023-12-12T02:00:35.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1053611008 bytes 2023-12-12T02:00:35.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1090099200 bytes 2023-12-12T02:00:35.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1119095808 bytes 2023-12-12T02:00:35.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1151886336 bytes 2023-12-12T02:00:35.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1152474112 bytes 2023-12-12T02:00:36.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1197064192 bytes 2023-12-12T02:00:36.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1241293824 bytes 2023-12-12T02:00:36.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1279827968 bytes 2023-12-12T02:00:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1364960256 bytes 2023-12-12T02:00:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1421118464 bytes 2023-12-12T02:00:37.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1458508800 bytes 2023-12-12T02:00:37.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1531479040 bytes 2023-12-12T02:00:37.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1566224384 bytes 2023-12-12T02:00:37.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1609879552 bytes 2023-12-12T02:00:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1674695680 bytes 2023-12-12T02:00:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1706588160 bytes 2023-12-12T02:00:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1713645568 bytes 2023-12-12T02:00:38.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1777870848 bytes 2023-12-12T02:00:38.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1850858496 bytes 2023-12-12T02:00:39.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_32 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_32 128M 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:39.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_32 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2023-12-12T02:00:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:00:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2023-12-12T02:00:40.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:40.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2023-12-12T02:00:40.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_32': 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13a89352dc9e 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13a89352dc9e 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:39 2023 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:39 2023 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:39 2023 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0373fde7-896e-4605-8ba0-e5ef5563088e 2023-12-12T02:00:40.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:40.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_32 2023-12-12T02:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:00:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:40.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:40.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 52935680 bytes 2023-12-12T02:00:40.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 67102720 bytes 2023-12-12T02:00:40.711 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 134429696 bytes 2023-12-12T02:00:40.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 136344576 bytes 2023-12-12T02:00:41.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 224656384 bytes 2023-12-12T02:00:41.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 274276352 bytes 2023-12-12T02:00:41.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 313438208 bytes 2023-12-12T02:00:41.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 363741184 bytes 2023-12-12T02:00:41.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 386738176 bytes 2023-12-12T02:00:42.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 460388352 bytes 2023-12-12T02:00:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 495026176 bytes 2023-12-12T02:00:42.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 535782400 bytes 2023-12-12T02:00:42.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 610590720 bytes 2023-12-12T02:00:43.056 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 661796864 bytes 2023-12-12T02:00:43.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 686144512 bytes 2023-12-12T02:00:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 719645696 bytes 2023-12-12T02:00:43.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 795589632 bytes 2023-12-12T02:00:43.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 815226880 bytes 2023-12-12T02:00:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 834051072 bytes 2023-12-12T02:00:44.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 867461120 bytes 2023-12-12T02:00:44.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 943063040 bytes 2023-12-12T02:00:44.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 968540160 bytes 2023-12-12T02:00:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1019572224 bytes 2023-12-12T02:00:44.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1085027328 bytes 2023-12-12T02:00:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1129038848 bytes 2023-12-12T02:00:45.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1177259008 bytes 2023-12-12T02:00:45.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1226803200 bytes 2023-12-12T02:00:45.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1252817920 bytes 2023-12-12T02:00:46.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1341278208 bytes 2023-12-12T02:00:46.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_33 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:46.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_33 128M 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:46.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_33 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:46.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2023-12-12T02:00:46.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:46.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_33': 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13b1e33d5d34 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13b1e33d5d34 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:46.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:46 2023 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:46 2023 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:46 2023 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 35e5deae-4560-4878-ab3c-ca16cbea0129 2023-12-12T02:00:46.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_33 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:00:46.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:00:46.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:46.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 54388736 bytes 2023-12-12T02:00:47.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 120655872 bytes 2023-12-12T02:00:47.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 232801280 bytes 2023-12-12T02:00:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 283991040 bytes 2023-12-12T02:00:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 290634752 bytes 2023-12-12T02:00:48.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 351994880 bytes 2023-12-12T02:00:48.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 383169536 bytes 2023-12-12T02:00:48.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 383769600 bytes 2023-12-12T02:00:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 384393216 bytes 2023-12-12T02:00:48.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 408579072 bytes 2023-12-12T02:00:48.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 488725504 bytes 2023-12-12T02:00:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 565474304 bytes 2023-12-12T02:00:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 604838912 bytes 2023-12-12T02:00:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 646969344 bytes 2023-12-12T02:00:49.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 647612416 bytes 2023-12-12T02:00:49.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 697058304 bytes 2023-12-12T02:00:50.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 774348800 bytes 2023-12-12T02:00:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 804807680 bytes 2023-12-12T02:00:50.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 834812928 bytes 2023-12-12T02:00:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 861553664 bytes 2023-12-12T02:00:50.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 935401472 bytes 2023-12-12T02:00:51.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 959945728 bytes 2023-12-12T02:00:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1083609088 bytes 2023-12-12T02:00:51.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1169752064 bytes 2023-12-12T02:00:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1205556224 bytes 2023-12-12T02:00:52.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1245400064 bytes 2023-12-12T02:00:52.121 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:00:52.125+0000 7fd7ab621640 -1 reset not still connected to 0x5574732fe4e0 2023-12-12T02:00:52.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1264439296 bytes 2023-12-12T02:00:52.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1345393664 bytes 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_34 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_34 128M 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:00:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:00:52.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_34 2023-12-12T02:00:52.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:00:52.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2023-12-12T02:00:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:00:53.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2023-12-12T02:00:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_34': 2023-12-12T02:00:53.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13baa7ae2c5 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13baa7ae2c5 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:00:53.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:52 2023 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:52 2023 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:52 2023 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 9d532316-c42a-45ca-a0b9-856d68476664 2023-12-12T02:00:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:00:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:00:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_34 2023-12-12T02:00:53.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:00:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:00:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:00:53.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:00:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T02:00:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:00:53.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:00:53.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 42876928 bytes 2023-12-12T02:00:53.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 101686272 bytes 2023-12-12T02:00:53.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 137621504 bytes 2023-12-12T02:00:53.967 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 200297472 bytes 2023-12-12T02:00:54.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 235220992 bytes 2023-12-12T02:00:54.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 266443776 bytes 2023-12-12T02:00:54.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 277828608 bytes 2023-12-12T02:00:54.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 314521600 bytes 2023-12-12T02:00:54.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 369852416 bytes 2023-12-12T02:00:55.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 433866752 bytes 2023-12-12T02:00:55.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 478652416 bytes 2023-12-12T02:00:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 526520320 bytes 2023-12-12T02:00:55.711 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 579067904 bytes 2023-12-12T02:00:55.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 601300992 bytes 2023-12-12T02:00:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 621274112 bytes 2023-12-12T02:00:56.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 695654400 bytes 2023-12-12T02:00:56.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 747845632 bytes 2023-12-12T02:00:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 788056064 bytes 2023-12-12T02:00:56.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 788673536 bytes 2023-12-12T02:00:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 863216640 bytes 2023-12-12T02:00:57.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 901477376 bytes 2023-12-12T02:00:57.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 926166016 bytes 2023-12-12T02:00:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 947168256 bytes 2023-12-12T02:00:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 992293888 bytes 2023-12-12T02:00:58.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1041963008 bytes 2023-12-12T02:00:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1058189312 bytes 2023-12-12T02:00:58.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1071444992 bytes 2023-12-12T02:00:58.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1103248384 bytes 2023-12-12T02:00:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1186663424 bytes 2023-12-12T02:00:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1187288064 bytes 2023-12-12T02:00:59.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1218985984 bytes 2023-12-12T02:00:59.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1254957056 bytes 2023-12-12T02:00:59.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1280883712 bytes 2023-12-12T02:00:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1309008896 bytes 2023-12-12T02:00:59.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1349766144 bytes 2023-12-12T02:01:00.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1369954304 bytes 2023-12-12T02:01:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1393928192 bytes 2023-12-12T02:01:00.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1394541568 bytes 2023-12-12T02:01:00.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1469414400 bytes 2023-12-12T02:01:00.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1532409856 bytes 2023-12-12T02:01:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1582938112 bytes 2023-12-12T02:01:01.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1617537024 bytes 2023-12-12T02:01:01.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1654277120 bytes 2023-12-12T02:01:01.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1664088064 bytes 2023-12-12T02:01:01.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1686223872 bytes 2023-12-12T02:01:01.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1729395712 bytes 2023-12-12T02:01:02.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1763812352 bytes 2023-12-12T02:01:02.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1816767488 bytes 2023-12-12T02:01:02.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1866625024 bytes 2023-12-12T02:01:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1885489152 bytes 2023-12-12T02:01:02.880 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1927404544 bytes 2023-12-12T02:01:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 1969683456 bytes 2023-12-12T02:01:03.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2023979008 bytes 2023-12-12T02:01:03.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2032378880 bytes 2023-12-12T02:01:03.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2070508544 bytes 2023-12-12T02:01:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2116911104 bytes 2023-12-12T02:01:03.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2170401792 bytes 2023-12-12T02:01:04.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2212336640 bytes 2023-12-12T02:01:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2283047936 bytes 2023-12-12T02:01:04.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2383606784 bytes 2023-12-12T02:01:04.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2388682752 bytes 2023-12-12T02:01:05.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2442102784 bytes 2023-12-12T02:01:05.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2476392448 bytes 2023-12-12T02:01:05.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2530285568 bytes 2023-12-12T02:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2578724864 bytes 2023-12-12T02:01:05.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2615133184 bytes 2023-12-12T02:01:06.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2618445824 bytes 2023-12-12T02:01:06.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_35 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_35 128M 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:06.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_35 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2023-12-12T02:01:06.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:06.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2023-12-12T02:01:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_35': 2023-12-12T02:01:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:06.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13c24739763d 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13c24739763d 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:06 2023 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:06 2023 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:06 2023 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: bf53f491-4b7f-44d9-bb31-ba260cb00dd6 2023-12-12T02:01:06.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:06.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_35 2023-12-12T02:01:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:01:06.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T02:01:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:06.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:06.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 25039872 bytes 2023-12-12T02:01:07.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 121927680 bytes 2023-12-12T02:01:07.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 151281664 bytes 2023-12-12T02:01:07.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 174020608 bytes 2023-12-12T02:01:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 190905344 bytes 2023-12-12T02:01:07.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 243918848 bytes 2023-12-12T02:01:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 302965760 bytes 2023-12-12T02:01:08.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 322755584 bytes 2023-12-12T02:01:08.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 402710528 bytes 2023-12-12T02:01:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 434034688 bytes 2023-12-12T02:01:08.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 454466560 bytes 2023-12-12T02:01:08.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 472516608 bytes 2023-12-12T02:01:09.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 506326016 bytes 2023-12-12T02:01:09.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 557430784 bytes 2023-12-12T02:01:09.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 664503296 bytes 2023-12-12T02:01:10.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 775755776 bytes 2023-12-12T02:01:10.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 797740032 bytes 2023-12-12T02:01:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 847430656 bytes 2023-12-12T02:01:10.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 874949632 bytes 2023-12-12T02:01:10.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 894961664 bytes 2023-12-12T02:01:11.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 998934528 bytes 2023-12-12T02:01:11.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1059142656 bytes 2023-12-12T02:01:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1096982528 bytes 2023-12-12T02:01:11.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1109053440 bytes 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_36 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:11.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_36 128M 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:11.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_36 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2023-12-12T02:01:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:01:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2023-12-12T02:01:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:12.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2023-12-12T02:01:12.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_36': 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13cba88a3491 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13cba88a3491 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:11 2023 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:11 2023 2023-12-12T02:01:12.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:11 2023 2023-12-12T02:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 11209fbb-fa2b-470e-83df-ef0a30ead77a 2023-12-12T02:01:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_36 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:01:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T02:01:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:12.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:12.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 31153152 bytes 2023-12-12T02:01:12.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 58631168 bytes 2023-12-12T02:01:12.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 157464576 bytes 2023-12-12T02:01:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 173216768 bytes 2023-12-12T02:01:13.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 232557568 bytes 2023-12-12T02:01:13.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 312577024 bytes 2023-12-12T02:01:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 346817536 bytes 2023-12-12T02:01:13.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 367389696 bytes 2023-12-12T02:01:13.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 401044480 bytes 2023-12-12T02:01:14.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 439553024 bytes 2023-12-12T02:01:14.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 463692800 bytes 2023-12-12T02:01:14.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 525767680 bytes 2023-12-12T02:01:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 561218560 bytes 2023-12-12T02:01:14.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 592077824 bytes 2023-12-12T02:01:15.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 671024128 bytes 2023-12-12T02:01:15.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 693911552 bytes 2023-12-12T02:01:15.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 713461760 bytes 2023-12-12T02:01:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 777698304 bytes 2023-12-12T02:01:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 791467008 bytes 2023-12-12T02:01:16.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 812638208 bytes 2023-12-12T02:01:16.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 865027072 bytes 2023-12-12T02:01:16.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 897790976 bytes 2023-12-12T02:01:16.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 908081152 bytes 2023-12-12T02:01:16.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 938970112 bytes 2023-12-12T02:01:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 985007104 bytes 2023-12-12T02:01:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1035537408 bytes 2023-12-12T02:01:17.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1119092736 bytes 2023-12-12T02:01:17.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1165782016 bytes 2023-12-12T02:01:17.857 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1206787072 bytes 2023-12-12T02:01:18.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1278259200 bytes 2023-12-12T02:01:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1299244032 bytes 2023-12-12T02:01:18.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1367223296 bytes 2023-12-12T02:01:18.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1399678976 bytes 2023-12-12T02:01:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1469028352 bytes 2023-12-12T02:01:19.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1517230080 bytes 2023-12-12T02:01:19.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1568812032 bytes 2023-12-12T02:01:19.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1629965312 bytes 2023-12-12T02:01:19.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1662846976 bytes 2023-12-12T02:01:19.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1706795008 bytes 2023-12-12T02:01:20.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1716213760 bytes 2023-12-12T02:01:20.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1766875136 bytes 2023-12-12T02:01:20.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1790870528 bytes 2023-12-12T02:01:20.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1791481856 bytes 2023-12-12T02:01:20.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1805049856 bytes 2023-12-12T02:01:20.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1825419264 bytes 2023-12-12T02:01:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1906938880 bytes 2023-12-12T02:01:21.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_37 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:21.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_37 128M 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:21.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_37 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:21.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2023-12-12T02:01:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:21.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2023-12-12T02:01:21.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_37': 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13d438de0f25 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13d438de0f25 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:21 2023 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:21 2023 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:21 2023 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 2e8a2d60-c409-430d-a9f4-c3c007ec57eb 2023-12-12T02:01:21.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:21.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_37 2023-12-12T02:01:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:01:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T02:01:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:21.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:22.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 118081536 bytes 2023-12-12T02:01:22.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 188617728 bytes 2023-12-12T02:01:22.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 189257728 bytes 2023-12-12T02:01:22.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 238366720 bytes 2023-12-12T02:01:23.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 334828544 bytes 2023-12-12T02:01:23.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 406701056 bytes 2023-12-12T02:01:23.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 449068032 bytes 2023-12-12T02:01:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 558340096 bytes 2023-12-12T02:01:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 605563904 bytes 2023-12-12T02:01:24.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 704171008 bytes 2023-12-12T02:01:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 771457024 bytes 2023-12-12T02:01:24.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 807605248 bytes 2023-12-12T02:01:25.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 835269632 bytes 2023-12-12T02:01:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 854740992 bytes 2023-12-12T02:01:25.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 892911616 bytes 2023-12-12T02:01:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 956032000 bytes 2023-12-12T02:01:25.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 1019388928 bytes 2023-12-12T02:01:25.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 1032422400 bytes 2023-12-12T02:01:26.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1072888832 bytes 2023-12-12T02:01:26.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1129497600 bytes 2023-12-12T02:01:26.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1130120192 bytes 2023-12-12T02:01:26.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1149315072 bytes 2023-12-12T02:01:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1243594752 bytes 2023-12-12T02:01:27.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1244205056 bytes 2023-12-12T02:01:27.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1266442240 bytes 2023-12-12T02:01:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1362661376 bytes 2023-12-12T02:01:27.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1419835392 bytes 2023-12-12T02:01:28.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1532886016 bytes 2023-12-12T02:01:28.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1579703296 bytes 2023-12-12T02:01:28.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1595606016 bytes 2023-12-12T02:01:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1637772288 bytes 2023-12-12T02:01:28.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1704056832 bytes 2023-12-12T02:01:29.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1761377280 bytes 2023-12-12T02:01:29.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1798403072 bytes 2023-12-12T02:01:29.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1812183040 bytes 2023-12-12T02:01:29.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1823241216 bytes 2023-12-12T02:01:29.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1844931584 bytes 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_38 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_38 128M 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:29.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_38 2023-12-12T02:01:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:29.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2023-12-12T02:01:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_38': 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13de4a7b4a80 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13de4a7b4a80 2023-12-12T02:01:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:29 2023 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:29 2023 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:29 2023 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 3fd2a8e9-d9a7-49d9-9916-b33d7e40559b 2023-12-12T02:01:30.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:30.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_38 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:30.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T02:01:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:30.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 21604352 bytes 2023-12-12T02:01:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 59226112 bytes 2023-12-12T02:01:30.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 79371264 bytes 2023-12-12T02:01:30.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 79974400 bytes 2023-12-12T02:01:30.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 108654592 bytes 2023-12-12T02:01:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 188008448 bytes 2023-12-12T02:01:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 245413888 bytes 2023-12-12T02:01:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 300617728 bytes 2023-12-12T02:01:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 325536768 bytes 2023-12-12T02:01:31.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 333238272 bytes 2023-12-12T02:01:32.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 350307328 bytes 2023-12-12T02:01:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 350922752 bytes 2023-12-12T02:01:32.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 408737792 bytes 2023-12-12T02:01:32.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 456391680 bytes 2023-12-12T02:01:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 485663744 bytes 2023-12-12T02:01:32.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 496430080 bytes 2023-12-12T02:01:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 541668352 bytes 2023-12-12T02:01:33.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 606166016 bytes 2023-12-12T02:01:33.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 643235840 bytes 2023-12-12T02:01:33.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 666412032 bytes 2023-12-12T02:01:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 680140800 bytes 2023-12-12T02:01:33.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 695560192 bytes 2023-12-12T02:01:34.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 750509056 bytes 2023-12-12T02:01:34.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 798779392 bytes 2023-12-12T02:01:34.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 807638016 bytes 2023-12-12T02:01:34.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 854924288 bytes 2023-12-12T02:01:34.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 901868544 bytes 2023-12-12T02:01:35.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 919768064 bytes 2023-12-12T02:01:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 992091136 bytes 2023-12-12T02:01:35.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1018737664 bytes 2023-12-12T02:01:35.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1019373568 bytes 2023-12-12T02:01:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1072869376 bytes 2023-12-12T02:01:36.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1104405504 bytes 2023-12-12T02:01:36.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1166614528 bytes 2023-12-12T02:01:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1210009600 bytes 2023-12-12T02:01:36.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1223111680 bytes 2023-12-12T02:01:36.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1297990656 bytes 2023-12-12T02:01:36.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1352535040 bytes 2023-12-12T02:01:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1422752768 bytes 2023-12-12T02:01:37.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1501202432 bytes 2023-12-12T02:01:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1557579776 bytes 2023-12-12T02:01:37.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1612524544 bytes 2023-12-12T02:01:38.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1629656064 bytes 2023-12-12T02:01:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:38.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_39 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:38.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_39 128M 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:38.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_39 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2023-12-12T02:01:38.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8190685338924114) > weight (0.25) 2023-12-12T02:01:38.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8431785543089071) > weight (0.25) 2023-12-12T02:01:38.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5194748381873644) > weight (0.25) 2023-12-12T02:01:38.991 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8767537397438935) > weight (0.25) 2023-12-12T02:01:38.991 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2023-12-12T02:01:39.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_39': 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13e95f4f0ce7 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13e95f4f0ce7 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:39.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:38 2023 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:38 2023 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:38 2023 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: bdb4862d-315a-44e3-8a44-de343b3940cd 2023-12-12T02:01:39.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_39 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:01:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:39.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T02:01:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:39.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:39.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 34383872 bytes 2023-12-12T02:01:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 86127616 bytes 2023-12-12T02:01:39.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 90138624 bytes 2023-12-12T02:01:39.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 115877888 bytes 2023-12-12T02:01:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 137744384 bytes 2023-12-12T02:01:40.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 194401280 bytes 2023-12-12T02:01:40.374 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.375+0000 7f5c51186640 -1 reset not still connected to 0x558c1f45a000 2023-12-12T02:01:40.374 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.375+0000 7f5c51186640 -1 reset not still connected to 0x558c2197f110 2023-12-12T02:01:40.375 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.379+0000 7f5c51186640 -1 reset not still connected to 0x558c23608b60 2023-12-12T02:01:40.375 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.379+0000 7f5c51186640 -1 reset not still connected to 0x558c2579cd00 2023-12-12T02:01:40.375 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.379+0000 7f5c51186640 -1 reset not still connected to 0x558c53a38ea0 2023-12-12T02:01:40.375 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:01:40.379+0000 7f5c51186640 -1 reset not still connected to 0x558c64a1a1a0 2023-12-12T02:01:40.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 220636160 bytes 2023-12-12T02:01:40.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 268702720 bytes 2023-12-12T02:01:40.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 304109568 bytes 2023-12-12T02:01:41.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 380943360 bytes 2023-12-12T02:01:41.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 417903616 bytes 2023-12-12T02:01:41.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 432153600 bytes 2023-12-12T02:01:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 524803072 bytes 2023-12-12T02:01:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 574288896 bytes 2023-12-12T02:01:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 624392192 bytes 2023-12-12T02:01:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 642178048 bytes 2023-12-12T02:01:42.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 642804736 bytes 2023-12-12T02:01:42.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 665976832 bytes 2023-12-12T02:01:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 688671744 bytes 2023-12-12T02:01:42.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 735557632 bytes 2023-12-12T02:01:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 741093376 bytes 2023-12-12T02:01:43.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 770491392 bytes 2023-12-12T02:01:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 821866496 bytes 2023-12-12T02:01:43.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 841252864 bytes 2023-12-12T02:01:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 902573056 bytes 2023-12-12T02:01:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 956174336 bytes 2023-12-12T02:01:44.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 984011776 bytes 2023-12-12T02:01:44.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1081381888 bytes 2023-12-12T02:01:44.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1108834304 bytes 2023-12-12T02:01:44.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1114019840 bytes 2023-12-12T02:01:44.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1150823424 bytes 2023-12-12T02:01:45.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1191526400 bytes 2023-12-12T02:01:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1258044416 bytes 2023-12-12T02:01:45.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1275144192 bytes 2023-12-12T02:01:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1324442624 bytes 2023-12-12T02:01:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1336030208 bytes 2023-12-12T02:01:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1397623808 bytes 2023-12-12T02:01:46.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1412771840 bytes 2023-12-12T02:01:46.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1509019648 bytes 2023-12-12T02:01:46.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1533429760 bytes 2023-12-12T02:01:46.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1552503808 bytes 2023-12-12T02:01:47.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1580007424 bytes 2023-12-12T02:01:47.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1651548160 bytes 2023-12-12T02:01:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1653672960 bytes 2023-12-12T02:01:47.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1699339264 bytes 2023-12-12T02:01:47.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1734264832 bytes 2023-12-12T02:01:48.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1765965824 bytes 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_40 2023-12-12T02:01:48.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_40 128M 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:48.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_40 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2023-12-12T02:01:48.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:01:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:48.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2023-12-12T02:01:48.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:48.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2023-12-12T02:01:48.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_40': 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13f11b7a4a0e 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13f11b7a4a0e 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:48 2023 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:48 2023 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:48 2023 2023-12-12T02:01:48.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0b428cbf-b8e2-4dfc-8c1f-e99c27ead9e9 2023-12-12T02:01:48.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_40 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:01:48.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:48.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T02:01:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:48.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 14256128 bytes 2023-12-12T02:01:49.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 52740096 bytes 2023-12-12T02:01:49.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 149123072 bytes 2023-12-12T02:01:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 208223232 bytes 2023-12-12T02:01:49.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 246152192 bytes 2023-12-12T02:01:50.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 246764544 bytes 2023-12-12T02:01:50.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 283674624 bytes 2023-12-12T02:01:50.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 332035072 bytes 2023-12-12T02:01:50.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 382302208 bytes 2023-12-12T02:01:50.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 443061248 bytes 2023-12-12T02:01:51.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 504694784 bytes 2023-12-12T02:01:51.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 543593472 bytes 2023-12-12T02:01:51.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 557511680 bytes 2023-12-12T02:01:51.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 639910912 bytes 2023-12-12T02:01:52.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 716763136 bytes 2023-12-12T02:01:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 753388544 bytes 2023-12-12T02:01:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 769505280 bytes 2023-12-12T02:01:52.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 847105024 bytes 2023-12-12T02:01:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 890692608 bytes 2023-12-12T02:01:53.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 930990080 bytes 2023-12-12T02:01:53.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 979279872 bytes 2023-12-12T02:01:53.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1019670528 bytes 2023-12-12T02:01:53.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1059789824 bytes 2023-12-12T02:01:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1078428672 bytes 2023-12-12T02:01:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1185990656 bytes 2023-12-12T02:01:54.282 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1237101568 bytes 2023-12-12T02:01:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1243868160 bytes 2023-12-12T02:01:54.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1254154240 bytes 2023-12-12T02:01:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1268052992 bytes 2023-12-12T02:01:54.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1299176448 bytes 2023-12-12T02:01:55.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1348740096 bytes 2023-12-12T02:01:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1408231424 bytes 2023-12-12T02:01:55.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1426159616 bytes 2023-12-12T02:01:55.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1452813312 bytes 2023-12-12T02:01:55.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1467017216 bytes 2023-12-12T02:01:55.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1500921856 bytes 2023-12-12T02:01:56.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1537930240 bytes 2023-12-12T02:01:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1543316480 bytes 2023-12-12T02:01:56.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1607183360 bytes 2023-12-12T02:01:56.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1624141824 bytes 2023-12-12T02:01:56.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_41 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:01:56.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_41 128M 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:01:56.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_41 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2023-12-12T02:01:56.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:01:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:01:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2023-12-12T02:01:56.979 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3522559707098726) > weight (0.25) 2023-12-12T02:01:56.980 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7047832431890955) > weight (0.25) 2023-12-12T02:01:56.980 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-12T02:01:56.980 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-12T02:01:56.980 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-12T02:01:56.981 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:01:56.987+0000 7f88b6ae8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44913) UID: 1000 2023-12-12T02:01:57.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:01:57.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2023-12-12T02:01:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_41': 2023-12-12T02:01:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:01:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 13fade5e94f2 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.13fade5e94f2 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:56 2023 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:56 2023 2023-12-12T02:01:57.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:56 2023 2023-12-12T02:01:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:01:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:01:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: aee6cea0-e960-4a62-b9b4-73a03085ac96 2023-12-12T02:01:57.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_41 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:01:57.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:01:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T02:01:57.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:01:57.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:01:57.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 13506560 bytes 2023-12-12T02:01:57.981 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-12T02:01:57.981 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:01:57.981 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-12T02:01:57.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-12T02:01:57.982 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-12T02:01:57.984 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-12T02:01:57.985 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2023-12-12T02:01:58.171 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T02:01:58.174+0000 7f2a08a08640 -1 reset not still connected to 0x55c66ba28f70 2023-12-12T02:01:58.172 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T02:01:58.174+0000 7f2a08a08640 -1 reset not still connected to 0x55c66d59a4e0 2023-12-12T02:01:58.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 85826560 bytes 2023-12-12T02:01:58.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 130541568 bytes 2023-12-12T02:01:58.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 159800320 bytes 2023-12-12T02:01:58.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 216324096 bytes 2023-12-12T02:01:59.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 290418688 bytes 2023-12-12T02:01:59.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 334292992 bytes 2023-12-12T02:01:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 394251264 bytes 2023-12-12T02:01:59.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 453198848 bytes 2023-12-12T02:01:59.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 505316352 bytes 2023-12-12T02:02:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 564979712 bytes 2023-12-12T02:02:00.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 610172928 bytes 2023-12-12T02:02:00.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 660347904 bytes 2023-12-12T02:02:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 710122496 bytes 2023-12-12T02:02:01.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 749336576 bytes 2023-12-12T02:02:01.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 774789120 bytes 2023-12-12T02:02:01.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 802300928 bytes 2023-12-12T02:02:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 848463872 bytes 2023-12-12T02:02:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 855424000 bytes 2023-12-12T02:02:01.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 929316864 bytes 2023-12-12T02:02:02.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 951219200 bytes 2023-12-12T02:02:02.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 957263872 bytes 2023-12-12T02:02:02.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1081500672 bytes 2023-12-12T02:02:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1111096320 bytes 2023-12-12T02:02:03.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1193819136 bytes 2023-12-12T02:02:03.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1254572032 bytes 2023-12-12T02:02:03.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1352184832 bytes 2023-12-12T02:02:03.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1415548928 bytes 2023-12-12T02:02:03.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1423852544 bytes 2023-12-12T02:02:04.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1524870144 bytes 2023-12-12T02:02:04.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1590777856 bytes 2023-12-12T02:02:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1615529984 bytes 2023-12-12T02:02:04.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1641560064 bytes 2023-12-12T02:02:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1711183872 bytes 2023-12-12T02:02:05.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1752695808 bytes 2023-12-12T02:02:05.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1783729152 bytes 2023-12-12T02:02:05.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1834175488 bytes 2023-12-12T02:02:05.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1880988672 bytes 2023-12-12T02:02:05.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1881857024 bytes 2023-12-12T02:02:06.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1901437952 bytes 2023-12-12T02:02:06.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1949153280 bytes 2023-12-12T02:02:06.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1970123776 bytes 2023-12-12T02:02:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 2007436288 bytes 2023-12-12T02:02:06.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_42 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_42 128M 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_42 2023-12-12T02:02:06.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:06.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2023-12-12T02:02:07.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_42': 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 140ca5c5f4a 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.140ca5c5f4a 2023-12-12T02:02:07.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:06 2023 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:06 2023 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:06 2023 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 82ea7790-1532-430b-aec5-73538850cd6e 2023-12-12T02:02:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_42 2023-12-12T02:02:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:02:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T02:02:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:07.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:07.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 4443136 bytes 2023-12-12T02:02:08.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 60818432 bytes 2023-12-12T02:02:08.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 110371840 bytes 2023-12-12T02:02:08.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 203548672 bytes 2023-12-12T02:02:08.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 251472896 bytes 2023-12-12T02:02:08.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 252098560 bytes 2023-12-12T02:02:09.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 263755776 bytes 2023-12-12T02:02:09.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 334560256 bytes 2023-12-12T02:02:09.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 391921664 bytes 2023-12-12T02:02:09.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 406359040 bytes 2023-12-12T02:02:09.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 455892992 bytes 2023-12-12T02:02:10.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 465152000 bytes 2023-12-12T02:02:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 478113792 bytes 2023-12-12T02:02:10.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 530580480 bytes 2023-12-12T02:02:10.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 560588800 bytes 2023-12-12T02:02:10.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 604519424 bytes 2023-12-12T02:02:10.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 651168768 bytes 2023-12-12T02:02:11.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 721932288 bytes 2023-12-12T02:02:11.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 802453504 bytes 2023-12-12T02:02:11.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 815413248 bytes 2023-12-12T02:02:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 847483904 bytes 2023-12-12T02:02:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 870902784 bytes 2023-12-12T02:02:12.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 896481280 bytes 2023-12-12T02:02:12.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 897125376 bytes 2023-12-12T02:02:12.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1037474816 bytes 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_43 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:02:12.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_43 128M 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:12.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_43 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2023-12-12T02:02:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:02:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:12.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2023-12-12T02:02:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:13.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_43': 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 14147fc84902 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.14147fc84902 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:13.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:12 2023 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:12 2023 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:12 2023 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a7f2974e-8b72-4697-9e38-a3cab091c892 2023-12-12T02:02:13.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_43 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:02:13.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T02:02:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:13.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:13.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 84698112 bytes 2023-12-12T02:02:13.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 100300800 bytes 2023-12-12T02:02:13.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 161838080 bytes 2023-12-12T02:02:14.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 300257280 bytes 2023-12-12T02:02:14.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 362853376 bytes 2023-12-12T02:02:14.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 403044352 bytes 2023-12-12T02:02:14.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 458651648 bytes 2023-12-12T02:02:15.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 482280448 bytes 2023-12-12T02:02:15.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 618846208 bytes 2023-12-12T02:02:15.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 665585664 bytes 2023-12-12T02:02:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 716172288 bytes 2023-12-12T02:02:16.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 765132800 bytes 2023-12-12T02:02:16.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 792008704 bytes 2023-12-12T02:02:16.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 803467264 bytes 2023-12-12T02:02:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 849627136 bytes 2023-12-12T02:02:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 930156544 bytes 2023-12-12T02:02:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 981948416 bytes 2023-12-12T02:02:17.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 1078310912 bytes 2023-12-12T02:02:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1083334656 bytes 2023-12-12T02:02:17.771 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1141275648 bytes 2023-12-12T02:02:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1163933696 bytes 2023-12-12T02:02:18.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1236794368 bytes 2023-12-12T02:02:18.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1336765440 bytes 2023-12-12T02:02:18.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1361426432 bytes 2023-12-12T02:02:18.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1391636480 bytes 2023-12-12T02:02:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1403992064 bytes 2023-12-12T02:02:19.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1464420352 bytes 2023-12-12T02:02:19.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1468634112 bytes 2023-12-12T02:02:19.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1524487168 bytes 2023-12-12T02:02:19.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1570672640 bytes 2023-12-12T02:02:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1601119232 bytes 2023-12-12T02:02:20.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1653424128 bytes 2023-12-12T02:02:20.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1676953600 bytes 2023-12-12T02:02:20.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1734539264 bytes 2023-12-12T02:02:20.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1777819648 bytes 2023-12-12T02:02:21.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1822222336 bytes 2023-12-12T02:02:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1829549056 bytes 2023-12-12T02:02:21.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1844264960 bytes 2023-12-12T02:02:21.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1891174400 bytes 2023-12-12T02:02:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1974584320 bytes 2023-12-12T02:02:21.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1993065472 bytes 2023-12-12T02:02:22.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 2022334464 bytes 2023-12-12T02:02:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 2044410880 bytes 2023-12-12T02:02:22.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2049641472 bytes 2023-12-12T02:02:22.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2052647936 bytes 2023-12-12T02:02:22.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2080403456 bytes 2023-12-12T02:02:22.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2117039104 bytes 2023-12-12T02:02:23.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2164612096 bytes 2023-12-12T02:02:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2191163392 bytes 2023-12-12T02:02:23.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2280914944 bytes 2023-12-12T02:02:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2304423936 bytes 2023-12-12T02:02:23.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2381506560 bytes 2023-12-12T02:02:24.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2382137344 bytes 2023-12-12T02:02:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:24.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_44 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_44 128M 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:24.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:24.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_44 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:24.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2023-12-12T02:02:25.109 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:25.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2023-12-12T02:02:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_44': 2023-12-12T02:02:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 141c21ed2bb8 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.141c21ed2bb8 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:24 2023 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:24 2023 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:24 2023 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0740d897-205d-4bac-ae1f-efbbf3092f25 2023-12-12T02:02:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_44 2023-12-12T02:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:25.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:02:25.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=9 2023-12-12T02:02:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:25.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:25.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 58323968 bytes 2023-12-12T02:02:25.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 89672704 bytes 2023-12-12T02:02:25.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 173596672 bytes 2023-12-12T02:02:26.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 205808640 bytes 2023-12-12T02:02:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 273622016 bytes 2023-12-12T02:02:26.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 348618752 bytes 2023-12-12T02:02:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 419547136 bytes 2023-12-12T02:02:27.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 464152576 bytes 2023-12-12T02:02:27.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 517975040 bytes 2023-12-12T02:02:27.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 564292608 bytes 2023-12-12T02:02:27.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 566276096 bytes 2023-12-12T02:02:27.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 580571136 bytes 2023-12-12T02:02:27.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 581172224 bytes 2023-12-12T02:02:27.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 621982720 bytes 2023-12-12T02:02:28.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 755107840 bytes 2023-12-12T02:02:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 765461504 bytes 2023-12-12T02:02:28.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 826422272 bytes 2023-12-12T02:02:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 830485504 bytes 2023-12-12T02:02:28.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 858793984 bytes 2023-12-12T02:02:29.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 892551168 bytes 2023-12-12T02:02:29.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 934330368 bytes 2023-12-12T02:02:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 975692800 bytes 2023-12-12T02:02:29.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 986025984 bytes 2023-12-12T02:02:29.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1011515392 bytes 2023-12-12T02:02:30.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1045486592 bytes 2023-12-12T02:02:30.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1084406784 bytes 2023-12-12T02:02:30.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1117191168 bytes 2023-12-12T02:02:30.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1169329152 bytes 2023-12-12T02:02:30.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1238702080 bytes 2023-12-12T02:02:31.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1286145024 bytes 2023-12-12T02:02:31.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1335996416 bytes 2023-12-12T02:02:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1377745920 bytes 2023-12-12T02:02:31.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1434194944 bytes 2023-12-12T02:02:31.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1445645312 bytes 2023-12-12T02:02:32.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1488861184 bytes 2023-12-12T02:02:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1524868096 bytes 2023-12-12T02:02:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1600036864 bytes 2023-12-12T02:02:32.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1659595776 bytes 2023-12-12T02:02:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1726707712 bytes 2023-12-12T02:02:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1791938560 bytes 2023-12-12T02:02:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1798707200 bytes 2023-12-12T02:02:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1835926528 bytes 2023-12-12T02:02:33.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1867578368 bytes 2023-12-12T02:02:33.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1992061952 bytes 2023-12-12T02:02:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:34.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_45 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_45 128M 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:34.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_45 2023-12-12T02:02:34.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:34.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2023-12-12T02:02:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:35.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2023-12-12T02:02:35.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_45': 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1424826ff8e1 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1424826ff8e1 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:34 2023 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:34 2023 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:34 2023 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: b49a1ee9-32a6-4049-9eef-ab8b4ccd2069 2023-12-12T02:02:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_45 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:02:35.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:35.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 58626048 bytes 2023-12-12T02:02:35.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 83609600 bytes 2023-12-12T02:02:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 86307840 bytes 2023-12-12T02:02:36.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 123436032 bytes 2023-12-12T02:02:36.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 186546176 bytes 2023-12-12T02:02:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 227090432 bytes 2023-12-12T02:02:36.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 243872768 bytes 2023-12-12T02:02:36.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 260465664 bytes 2023-12-12T02:02:37.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 281192448 bytes 2023-12-12T02:02:37.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 341406720 bytes 2023-12-12T02:02:37.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 357984256 bytes 2023-12-12T02:02:37.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 361392128 bytes 2023-12-12T02:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 390794240 bytes 2023-12-12T02:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 473838592 bytes 2023-12-12T02:02:38.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 511846400 bytes 2023-12-12T02:02:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 557846528 bytes 2023-12-12T02:02:38.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 634602496 bytes 2023-12-12T02:02:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 679680000 bytes 2023-12-12T02:02:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 689684480 bytes 2023-12-12T02:02:39.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 732490752 bytes 2023-12-12T02:02:39.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 774868992 bytes 2023-12-12T02:02:39.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 804659200 bytes 2023-12-12T02:02:39.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 867293184 bytes 2023-12-12T02:02:40.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 921070592 bytes 2023-12-12T02:02:40.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 954544128 bytes 2023-12-12T02:02:40.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 979192832 bytes 2023-12-12T02:02:40.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 994733056 bytes 2023-12-12T02:02:40.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1012452352 bytes 2023-12-12T02:02:40.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1037552640 bytes 2023-12-12T02:02:41.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1072890880 bytes 2023-12-12T02:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_46 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_46 128M 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:41.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_46 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2023-12-12T02:02:42.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_46': 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 142cf45ab19f 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.142cf45ab19f 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:42.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:41 2023 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:41 2023 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:41 2023 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 1e818293-2177-4212-8304-5c285074449e 2023-12-12T02:02:42.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_46 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:02:42.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:42.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T02:02:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:42.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:42.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 52645888 bytes 2023-12-12T02:02:42.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 109492224 bytes 2023-12-12T02:02:42.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 184654848 bytes 2023-12-12T02:02:43.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 259481600 bytes 2023-12-12T02:02:43.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 302301184 bytes 2023-12-12T02:02:43.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 302925824 bytes 2023-12-12T02:02:43.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 358023168 bytes 2023-12-12T02:02:43.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 395537408 bytes 2023-12-12T02:02:44.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 435985408 bytes 2023-12-12T02:02:44.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 509256704 bytes 2023-12-12T02:02:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 540275712 bytes 2023-12-12T02:02:44.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 559546368 bytes 2023-12-12T02:02:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 650323968 bytes 2023-12-12T02:02:45.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 658460672 bytes 2023-12-12T02:02:45.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 707935232 bytes 2023-12-12T02:02:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 755130368 bytes 2023-12-12T02:02:45.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 798780416 bytes 2023-12-12T02:02:45.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 853284864 bytes 2023-12-12T02:02:46.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 975240192 bytes 2023-12-12T02:02:46.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1023836160 bytes 2023-12-12T02:02:46.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1046867968 bytes 2023-12-12T02:02:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1064313856 bytes 2023-12-12T02:02:47.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1137126400 bytes 2023-12-12T02:02:47.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1146509312 bytes 2023-12-12T02:02:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1248416768 bytes 2023-12-12T02:02:47.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1268869120 bytes 2023-12-12T02:02:47.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1276034048 bytes 2023-12-12T02:02:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1281667072 bytes 2023-12-12T02:02:48.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1301689344 bytes 2023-12-12T02:02:48.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1306631168 bytes 2023-12-12T02:02:48.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1402134528 bytes 2023-12-12T02:02:48.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1436429312 bytes 2023-12-12T02:02:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1508501504 bytes 2023-12-12T02:02:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1509109760 bytes 2023-12-12T02:02:49.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1555486720 bytes 2023-12-12T02:02:49.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1574483968 bytes 2023-12-12T02:02:49.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1608237056 bytes 2023-12-12T02:02:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1648531456 bytes 2023-12-12T02:02:50.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1710055424 bytes 2023-12-12T02:02:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_47 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_47 128M 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_47 2023-12-12T02:02:50.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:50.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2023-12-12T02:02:51.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:51.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2023-12-12T02:02:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_47': 2023-12-12T02:02:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:51.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 14364d7ed2fb 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.14364d7ed2fb 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:50 2023 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:50 2023 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:50 2023 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:51.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: f7b0d429-ffe2-491a-861c-6f5dbcb6ae73 2023-12-12T02:02:51.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_47 2023-12-12T02:02:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:02:51.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=7 2023-12-12T02:02:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 62870528 bytes 2023-12-12T02:02:51.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 88690688 bytes 2023-12-12T02:02:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 129879040 bytes 2023-12-12T02:02:52.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 176464896 bytes 2023-12-12T02:02:52.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 243311616 bytes 2023-12-12T02:02:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 277116928 bytes 2023-12-12T02:02:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 286878720 bytes 2023-12-12T02:02:52.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 326567936 bytes 2023-12-12T02:02:52.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 373406720 bytes 2023-12-12T02:02:53.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 392287232 bytes 2023-12-12T02:02:53.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 416380928 bytes 2023-12-12T02:02:53.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 447243264 bytes 2023-12-12T02:02:53.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 487520256 bytes 2023-12-12T02:02:53.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 510988288 bytes 2023-12-12T02:02:53.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 551751680 bytes 2023-12-12T02:02:54.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 595954688 bytes 2023-12-12T02:02:54.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 606089216 bytes 2023-12-12T02:02:54.510 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 622050304 bytes 2023-12-12T02:02:54.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 665746432 bytes 2023-12-12T02:02:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 708849664 bytes 2023-12-12T02:02:55.056 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 732624896 bytes 2023-12-12T02:02:55.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 788990976 bytes 2023-12-12T02:02:55.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 802323456 bytes 2023-12-12T02:02:55.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 839595008 bytes 2023-12-12T02:02:55.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 922270720 bytes 2023-12-12T02:02:56.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 953747456 bytes 2023-12-12T02:02:56.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 969469952 bytes 2023-12-12T02:02:56.407 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1011260416 bytes 2023-12-12T02:02:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1027550208 bytes 2023-12-12T02:02:56.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1084445696 bytes 2023-12-12T02:02:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1087655936 bytes 2023-12-12T02:02:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1126537216 bytes 2023-12-12T02:02:57.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1187683328 bytes 2023-12-12T02:02:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1263816704 bytes 2023-12-12T02:02:57.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1307330560 bytes 2023-12-12T02:02:58.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1375406080 bytes 2023-12-12T02:02:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:02:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:02:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:02:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_48 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_48 128M 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:02:58.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_48 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2023-12-12T02:02:58.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:02:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:02:58.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2023-12-12T02:02:59.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:02:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_48': 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 143e5f0bdfff 2023-12-12T02:02:59.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.143e5f0bdfff 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:58 2023 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:58 2023 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:58 2023 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: dea4476e-358a-4185-8c03-a70bae8584fb 2023-12-12T02:02:59.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_48 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:02:59.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:02:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:02:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:02:59.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:02:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 39180288 bytes 2023-12-12T02:02:59.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 57603072 bytes 2023-12-12T02:02:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 117177344 bytes 2023-12-12T02:02:59.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 148652032 bytes 2023-12-12T02:03:00.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 180667392 bytes 2023-12-12T02:03:00.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 253847552 bytes 2023-12-12T02:03:00.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 286723072 bytes 2023-12-12T02:03:00.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 321638400 bytes 2023-12-12T02:03:00.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 349438976 bytes 2023-12-12T02:03:01.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 361211904 bytes 2023-12-12T02:03:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 464913408 bytes 2023-12-12T02:03:01.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 526535680 bytes 2023-12-12T02:03:01.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 555443200 bytes 2023-12-12T02:03:02.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 659941376 bytes 2023-12-12T02:03:02.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 660571136 bytes 2023-12-12T02:03:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 700743680 bytes 2023-12-12T02:03:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 701327360 bytes 2023-12-12T02:03:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 739899392 bytes 2023-12-12T02:03:02.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 788436992 bytes 2023-12-12T02:03:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 845061120 bytes 2023-12-12T02:03:03.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 878139392 bytes 2023-12-12T02:03:03.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 934131712 bytes 2023-12-12T02:03:03.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 969501696 bytes 2023-12-12T02:03:03.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1014495232 bytes 2023-12-12T02:03:04.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1090203648 bytes 2023-12-12T02:03:04.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1145337856 bytes 2023-12-12T02:03:04.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1194169344 bytes 2023-12-12T02:03:04.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1272841216 bytes 2023-12-12T02:03:04.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1279216640 bytes 2023-12-12T02:03:04.992 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T02:03:04.992 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T02:03:04.992 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T02:03:04.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:03:04.997+0000 7fb454183640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 50473) UID: 1000 2023-12-12T02:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:03:05.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_49 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_49 128M 2023-12-12T02:03:05.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:05.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_49 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:03:05.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2023-12-12T02:03:06.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:03:06.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2023-12-12T02:03:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_49': 2023-12-12T02:03:06.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 14464f38990a 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.14464f38990a 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:03:05 2023 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:03:05 2023 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:03:05 2023 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:03:06.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 50176e92-488b-4d3d-8b4d-6f801bfbf83c 2023-12-12T02:03:06.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:03:06.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:03:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_49 2023-12-12T02:03:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:03:06.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:03:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=8 2023-12-12T02:03:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:03:06.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:03:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 640000 bytes 2023-12-12T02:03:06.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 20613120 bytes 2023-12-12T02:03:06.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 68406272 bytes 2023-12-12T02:03:06.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 77731840 bytes 2023-12-12T02:03:06.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 78342144 bytes 2023-12-12T02:03:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 117064704 bytes 2023-12-12T02:03:07.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 148288512 bytes 2023-12-12T02:03:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 176201728 bytes 2023-12-12T02:03:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 220233728 bytes 2023-12-12T02:03:07.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 275970048 bytes 2023-12-12T02:03:08.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 361457664 bytes 2023-12-12T02:03:08.170 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:03:08.173+0000 7f5c51186640 -1 reset not still connected to 0x558c60f548f0 2023-12-12T02:03:08.171 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:03:08.177+0000 7f5c51186640 -1 reset not still connected to 0x558c625d5860 2023-12-12T02:03:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 452712448 bytes 2023-12-12T02:03:08.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 514993152 bytes 2023-12-12T02:03:08.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 539166720 bytes 2023-12-12T02:03:09.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 562319360 bytes 2023-12-12T02:03:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 610814976 bytes 2023-12-12T02:03:09.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 626558976 bytes 2023-12-12T02:03:09.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 645837824 bytes 2023-12-12T02:03:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 673890304 bytes 2023-12-12T02:03:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 704342016 bytes 2023-12-12T02:03:10.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 736108544 bytes 2023-12-12T02:03:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 760003584 bytes 2023-12-12T02:03:10.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 762041344 bytes 2023-12-12T02:03:10.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 816507904 bytes 2023-12-12T02:03:10.688 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 841735168 bytes 2023-12-12T02:03:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 940203008 bytes 2023-12-12T02:03:11.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1022783488 bytes 2023-12-12T02:03:11.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1090352128 bytes 2023-12-12T02:03:11.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1116069888 bytes 2023-12-12T02:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1151915008 bytes 2023-12-12T02:03:11.992 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T02:03:11.993 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:03:11.993 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T02:03:11.993 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T02:03:11.993 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T02:03:11.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T02:03:11.996 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-12T02:03:12.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1215171584 bytes 2023-12-12T02:03:12.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1279116288 bytes 2023-12-12T02:03:12.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1314244608 bytes 2023-12-12T02:03:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1354981376 bytes 2023-12-12T02:03:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1410807808 bytes 2023-12-12T02:03:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1431321600 bytes 2023-12-12T02:03:13.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1455453184 bytes 2023-12-12T02:03:13.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1483201536 bytes 2023-12-12T02:03:13.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1514020864 bytes 2023-12-12T02:03:13.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1550162944 bytes 2023-12-12T02:03:13.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1585105920 bytes 2023-12-12T02:03:14.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1642972160 bytes 2023-12-12T02:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:03:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_50 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n snapshot ] 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_50 128M 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 128M ] 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=128M 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:03:14.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering mirror/image_50 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ is_pool_mirror_mode_image mirror 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ image = image ] 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2023-12-12T02:03:14.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:03:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:03:14.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2023-12-12T02:03:15.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:03:15.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2023-12-12T02:03:15.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_50': 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1454c782dc9a 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1454c782dc9a 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:03:14 2023 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:03:14 2023 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:03:14 2023 2023-12-12T02:03:15.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:03:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:03:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 7a672f15-d393-4344-a316-8df164a5da70 2023-12-12T02:03:15.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n ] 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_50 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:03:15.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T02:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:03:15.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:03:15.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 615424 bytes 2023-12-12T02:03:15.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 34383872 bytes 2023-12-12T02:03:15.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 57205760 bytes 2023-12-12T02:03:15.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 69873664 bytes 2023-12-12T02:03:16.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 141829120 bytes 2023-12-12T02:03:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 173124608 bytes 2023-12-12T02:03:16.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 193839104 bytes 2023-12-12T02:03:16.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 228781056 bytes 2023-12-12T02:03:16.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 272108544 bytes 2023-12-12T02:03:16.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 284051456 bytes 2023-12-12T02:03:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 307703808 bytes 2023-12-12T02:03:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 322903040 bytes 2023-12-12T02:03:17.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 348531712 bytes 2023-12-12T02:03:17.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 436589568 bytes 2023-12-12T02:03:17.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 437208064 bytes 2023-12-12T02:03:18.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 462696448 bytes 2023-12-12T02:03:18.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 492721152 bytes 2023-12-12T02:03:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 552418304 bytes 2023-12-12T02:03:18.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 625868800 bytes 2023-12-12T02:03:19.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 630855680 bytes 2023-12-12T02:03:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 646498304 bytes 2023-12-12T02:03:19.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 670478336 bytes 2023-12-12T02:03:19.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 679143424 bytes 2023-12-12T02:03:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 699258880 bytes 2023-12-12T02:03:19.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 738051072 bytes 2023-12-12T02:03:20.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 780868608 bytes 2023-12-12T02:03:20.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 823006208 bytes 2023-12-12T02:03:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 823607296 bytes 2023-12-12T02:03:20.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 876000256 bytes 2023-12-12T02:03:20.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 885510144 bytes 2023-12-12T02:03:20.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 929870848 bytes 2023-12-12T02:03:21.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 934198272 bytes 2023-12-12T02:03:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 948034560 bytes 2023-12-12T02:03:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 994076672 bytes 2023-12-12T02:03:21.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1110319104 bytes 2023-12-12T02:03:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1150809088 bytes 2023-12-12T02:03:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1217000448 bytes 2023-12-12T02:03:22.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1237643264 bytes 2023-12-12T02:03:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1353452544 bytes 2023-12-12T02:03:22.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1402800128 bytes 2023-12-12T02:03:22.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1412618240 bytes 2023-12-12T02:03:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1413243904 bytes 2023-12-12T02:03:23.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1462087680 bytes 2023-12-12T02:03:23.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1464017920 bytes 2023-12-12T02:03:23.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1464633344 bytes 2023-12-12T02:03:23.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1483326464 bytes 2023-12-12T02:03:23.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1529589760 bytes 2023-12-12T02:03:24.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1580544000 bytes 2023-12-12T02:03:24.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1647903744 bytes 2023-12-12T02:03:24.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1648490496 bytes 2023-12-12T02:03:24.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1699578880 bytes 2023-12-12T02:03:24.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 1704120320 bytes 2023-12-12T02:03:24.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 1726354432 bytes 2023-12-12T02:03:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 1753609216 bytes 2023-12-12T02:03:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 1793888256 bytes 2023-12-12T02:03:25.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 1872286720 bytes 2023-12-12T02:03:25.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 1886082048 bytes 2023-12-12T02:03:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 1894986752 bytes 2023-12-12T02:03:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 1976666112 bytes 2023-12-12T02:03:26.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2014813184 bytes 2023-12-12T02:03:26.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2057719808 bytes 2023-12-12T02:03:26.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2087342080 bytes 2023-12-12T02:03:27.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2087977984 bytes 2023-12-12T02:03:27.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2136548352 bytes 2023-12-12T02:03:27.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2148446208 bytes 2023-12-12T02:03:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2176813056 bytes 2023-12-12T02:03:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2230537216 bytes 2023-12-12T02:03:28.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2292669440 bytes 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_images cluster2 mirror 50 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:28.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_count=50 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local count 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_count=0 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:03:28.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:03:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:03:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-12T02:03:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + cut -d -f 2 2023-12-12T02:03:28.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep images: 2023-12-12T02:03:28.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count=50 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 50 = 50 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_healthy cluster2 mirror 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state 2023-12-12T02:03:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:03:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:03:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-12T02:03:28.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep image health: 2023-12-12T02:03:28.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cut -d -f 3 2023-12-12T02:03:28.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:03:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=WARNING 2023-12-12T02:03:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test WARNING = ERROR 2023-12-12T02:03:28.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test WARNING = OK 2023-12-12T02:03:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 2 -ne 1 2023-12-12T02:03:28.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 30 2023-12-12T02:03:33.106 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:03:33.109+0000 7fd7ab621640 -1 reset not still connected to 0x557462e90a90 2023-12-12T02:03:33.107 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:03:33.109+0000 7fd7ab621640 -1 reset not still connected to 0x557465b1e270 2023-12-12T02:03:42.985 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9122493543807694) > weight (0.25) 2023-12-12T02:03:42.985 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T02:03:42.986 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T02:03:42.986 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-12T02:03:42.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:03:42.992+0000 7f557f2c0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 51354) UID: 1000 2023-12-12T02:03:51.987 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T02:03:51.987 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:03:51.987 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T02:03:51.988 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T02:03:51.988 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T02:03:51.991 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T02:03:51.991 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-12T02:03:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror mirror pool status 2023-12-12T02:03:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep image health: 2023-12-12T02:03:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cut -d -f 3 2023-12-12T02:03:58.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=OK 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = ERROR 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = OK 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_images cluster1 mirror 50 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_count=50 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local count 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_count=0 2023-12-12T02:03:58.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:03:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:03:58.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:03:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-12T02:03:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep images: 2023-12-12T02:03:58.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cut -d -f 2 2023-12-12T02:03:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:03:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count=50 2023-12-12T02:03:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 50 = 50 2023-12-12T02:03:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_healthy cluster1 mirror 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state 2023-12-12T02:03:58.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:03:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:03:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-12T02:03:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep image health: 2023-12-12T02:03:58.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:cut -d -f 3 2023-12-12T02:03:58.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:03:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=WARNING 2023-12-12T02:03:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test WARNING = ERROR 2023-12-12T02:03:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test WARNING = OK 2023-12-12T02:03:58.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 2 -ne 1 2023-12-12T02:03:58.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 30 2023-12-12T02:04:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd+ --cluster cluster1 -p mirror mirror pool status 2023-12-12T02:04:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep image health: 2023-12-12T02:04:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cut -d -f 3 2023-12-12T02:04:28.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=OK 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = ERROR 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = OK 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: compare many images 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T02:04:28.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T02:04:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 02:04:28 TEST: compare many images 2023-12-12T02:04:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 02:04:28 TEST: compare many images 2023-12-12T02:04:28.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 50 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_1 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_1 snap 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:28.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:04:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2023-12-12T02:04:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:29.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:29.996 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4236746171246528) > weight (0.25) 2023-12-12T02:04:29.997 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T02:04:29.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T02:04:29.997 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2023-12-12T02:04:29.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:04:30.000+0000 7f0e25278640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49731) UID: 1000 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:04:30.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_1 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:04:30.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep+ 0 2023-12-12T02:04:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:04:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:04:30.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:30.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:30.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:04:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:30.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2023-12-12T02:04:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:30.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:04:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T02:04:30.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T02:04:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:04:30.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:30.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_1.status 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_1.status 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.status 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_1.status 2023-12-12T02:04:30.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_1.status 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:30.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2023-12-12T02:04:31.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 119 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.status 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.status 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2023-12-12T02:04:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:31.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:04:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=119 2023-12-12T02:04:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:04:31.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-12T02:04:31.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:31.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 119 = 2023-12-12T02:04:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-12T02:04:31.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:31.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 119 = 2023-12-12T02:04:31.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:04:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:32.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.status 2023-12-12T02:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2023-12-12T02:04:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:32.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=119 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 119 = 119 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:32.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:33.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2023-12-12T02:04:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_1': 2023-12-12T02:04:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:04:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:04:33.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de9b77c2e 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de9b77c2e 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:14 2023 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:14 2023 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:14 2023 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: b834df41-6596-47b9-95f6-b6243a57f231 2023-12-12T02:04:33.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_1 snap 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.export 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.export 2023-12-12T02:04:33.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.export 2023-12-12T02:04:33.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.export 2023-12-12T02:04:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.export 2023-12-12T02:04:34.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:34.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.export 2023-12-12T02:04:34.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_1.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_1.export 2023-12-12T02:04:34.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_2 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_2 snap 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2023-12-12T02:04:35.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:35.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2023-12-12T02:04:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:36.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_2 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:04:36.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ greplocal -i instance=cluster1 state 2023-12-12T02:04:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:04:36.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:04:36.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:36.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:04:36.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:36.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2023-12-12T02:04:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:no valid command found; 2 closest matches: 2023-12-12T02:04:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_21 2023-12-12T02:04:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_24 2023-12-12T02:04:36.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:36.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:36.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:04:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:36.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2023-12-12T02:04:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:04:36.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:36.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2023-12-12T02:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:36.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:04:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:04:36.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:04:36.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:04:36.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:36.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_2.status 2023-12-12T02:04:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_2.status 2023-12-12T02:04:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:36.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.status 2023-12-12T02:04:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_2.status 2023-12-12T02:04:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_2.status 2023-12-12T02:04:36.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:36.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2023-12-12T02:04:37.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 121 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.status 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.status 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2023-12-12T02:04:37.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:37.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:04:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=121 2023-12-12T02:04:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:37.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:04:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:37.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-12T02:04:37.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:37.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=69 2023-12-12T02:04:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 121 = 69 2023-12-12T02:04:37.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-12T02:04:37.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:38.024 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:38.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 121 = 2023-12-12T02:04:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.status 2023-12-12T02:04:38.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2023-12-12T02:04:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:38.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=121 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 121 = 121 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:38.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:38.998 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T02:04:38.998 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:04:38.999 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T02:04:38.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T02:04:38.999 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T02:04:39.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T02:04:39.002 INFO:tasks.rbd_mirror_thrash:waiting for 115 secs before thrashing 2023-12-12T02:04:39.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_2': 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 11cdf7d6d124 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.11cdf7d6d124 2023-12-12T02:04:39.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:25 2023 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:25 2023 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:25 2023 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d1652be7-048c-4b97-9aef-5e3f0ab7f525 2023-12-12T02:04:39.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_2 snap 2023-12-12T02:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:04:39.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.export 2023-12-12T02:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.export 2023-12-12T02:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.export 2023-12-12T02:04:39.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.export 2023-12-12T02:04:40.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:40.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.export 2023-12-12T02:04:40.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:40.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.export 2023-12-12T02:04:40.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_2.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_2.export 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_3 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_3 snap 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:40.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:40.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:41.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:41.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:04:42.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:04:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_3 2023-12-12T02:04:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:04:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:04:42.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:04:42.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:42.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:42.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:42.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2023-12-12T02:04:42.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:04:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:04:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:04:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:04:42.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:42.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_3.status 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_3.status 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.status 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_3.status 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_3.status 2023-12-12T02:04:42.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:42.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2023-12-12T02:04:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 123 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.status 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.status 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2023-12-12T02:04:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:43.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=123 2023-12-12T02:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:43.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-12T02:04:43.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:43.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=70 2023-12-12T02:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 123 = 70 2023-12-12T02:04:43.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:04:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:43.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-12T02:04:43.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:44.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 123 = 2023-12-12T02:04:44.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:04:44.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.status 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2023-12-12T02:04:44.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:44.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=123 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 123 = 123 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:44.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:45.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2023-12-12T02:04:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_3': 2023-12-12T02:04:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:04:45.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b613a9e5b 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b613a9e5b 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:30 2023 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:30 2023 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:30 2023 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:04:45.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 489d4631-b2bd-47cc-9fa0-9d572c181d98 2023-12-12T02:04:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:04:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_3 snap 2023-12-12T02:04:45.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.export 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.export 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.export 2023-12-12T02:04:45.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.export 2023-12-12T02:04:46.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:46.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.export 2023-12-12T02:04:46.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:46.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.export 2023-12-12T02:04:46.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_3.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_3.export 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_4 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_4 snap 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:46.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2023-12-12T02:04:46.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:46.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:04:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1+ rbd mirror status mirror/image_4 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:04:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:47.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:47.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:04:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:47.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2023-12-12T02:04:47.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:no valid command found; 1 closest matches: 2023-12-12T02:04:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_44 2023-12-12T02:04:47.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:47.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:47.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:47.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:04:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:47.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2023-12-12T02:04:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2023-12-12T02:04:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:04:47.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:47.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:47.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:47.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:04:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:48.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:48.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2023-12-12T02:04:48.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:no valid command found; 3 closest matches: 2023-12-12T02:04:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_40 2023-12-12T02:04:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_42 2023-12-12T02:04:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_46 2023-12-12T02:04:48.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_4 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:04:48.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:48.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:48.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:04:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_4 2023-12-12T02:04:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:48.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:04:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:04:48.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:04:48.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:04:48.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_4.status 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_4.status 2023-12-12T02:04:48.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:04:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.status 2023-12-12T02:04:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_4.status 2023-12-12T02:04:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_4.status 2023-12-12T02:04:48.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:48.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2023-12-12T02:04:48.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 125 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.status 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.status 2023-12-12T02:04:48.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2023-12-12T02:04:48.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:04:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=125 2023-12-12T02:04:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:48.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:04:49.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-12T02:04:49.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:49.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=71 2023-12-12T02:04:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 125 = 71 2023-12-12T02:04:49.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:04:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:49.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-12T02:04:49.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:49.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 125 = 2023-12-12T02:04:49.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-12T02:04:50.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:50.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:50.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 125 = 2023-12-12T02:04:50.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606e1de60d0 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606e1de7450 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606e4834750 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606e4835040 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606e48356c0 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606dcd255f0 2023-12-12T02:04:51.528 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.521+0000 7f21394c6640 -1 reset not still connected to 0x5606dcd25ad0 2023-12-12T02:04:51.555 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.557+0000 7f21394c6640 -1 reset not still connected to 0x5606e1de7450 2023-12-12T02:04:51.556 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.557+0000 7f21394c6640 -1 reset not still connected to 0x5606e4834750 2023-12-12T02:04:51.556 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.557+0000 7f21394c6640 -1 reset not still connected to 0x5606e4835040 2023-12-12T02:04:51.556 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:04:51.557+0000 7f21394c6640 -1 reset not still connected to 0x5606e48356c0 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.status 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2023-12-12T02:04:52.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:52.111 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=125 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 125 = 125 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:52.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2023-12-12T02:04:53.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_4': 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 125e97ccc2ab 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.125e97ccc2ab 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:46 2023 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:46 2023 2023-12-12T02:04:53.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:46 2023 2023-12-12T02:04:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:04:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:04:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:04:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 6fcc8ecf-2681-4d44-82de-245f7523ff13 2023-12-12T02:04:53.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_4 snap 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.export 2023-12-12T02:04:53.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.export 2023-12-12T02:04:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.export 2023-12-12T02:04:53.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.export 2023-12-12T02:04:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:53.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.export 2023-12-12T02:04:53.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:04:53.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.export 2023-12-12T02:04:53.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_4.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_4.export 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_5 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_5 snap 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:04:53.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:04:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2023-12-12T02:04:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:54.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:04:54.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:04:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2023-12-12T02:04:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_5 2023-12-12T02:04:55.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:04:55.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:04:55.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:04:55.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:04:55.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:04:55.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:04:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:04:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2023-12-12T02:04:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:no valid command found; 1 closest matches: 2023-12-12T02:04:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status mirror/image_50 2023-12-12T02:04:55.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:55.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:04:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:04:55.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:04:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:04:55.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2023-12-12T02:04:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-12T02:04:55.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:04:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:55.436 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:04:55.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:04:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:04:55.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2023-12-12T02:04:55.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2023-12-12T02:04:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:04:55.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:55.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:04:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:04:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:04:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:04:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:04:55.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:04:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:04:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2023-12-12T02:04:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:04:55.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:04:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:04:55.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:55.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_5.status 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_5.status 2023-12-12T02:04:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.status 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_5.status 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + sed -e s|[/ ]|_|g 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo cluster1-mirror-image_5.status 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:55.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2023-12-12T02:04:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:55.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2023-12-12T02:04:56.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 127 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.status 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.status 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2023-12-12T02:04:56.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:56.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:04:56.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=127 2023-12-12T02:04:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:56.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:04:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:57.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-12T02:04:57.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:57.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=72 2023-12-12T02:04:57.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 127 = 72 2023-12-12T02:04:57.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-12T02:04:57.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:57.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 127 = 2023-12-12T02:04:57.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-12T02:04:58.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:04:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:04:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:04:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:04:58.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:04:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:04:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:04:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 127 = 2023-12-12T02:04:58.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.status 2023-12-12T02:05:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2023-12-12T02:05:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=127 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 127 = 127 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:00.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:01.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_5': 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1296221b2b2d 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1296221b2b2d 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:55:52 2023 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:55:52 2023 2023-12-12T02:05:01.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:55:52 2023 2023-12-12T02:05:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c945b710-ea44-42b4-9616-704aab1766b4 2023-12-12T02:05:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:01.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_5 snap 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.export 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.export 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.export 2023-12-12T02:05:01.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.export 2023-12-12T02:05:01.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:01.406 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.export 2023-12-12T02:05:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:01.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.export 2023-12-12T02:05:01.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_5.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_5.export 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_6 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_6 snap 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:01.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:01.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:01.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:02.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_6+ 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:03.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:03.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2023-12-12T02:05:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:03.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:03.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2023-12-12T02:05:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:03.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:03.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:03.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_6.status 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_6.status 2023-12-12T02:05:03.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.status 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_6.status 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_6.status 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:03.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2023-12-12T02:05:04.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 129 2023-12-12T02:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.status 2023-12-12T02:05:04.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.status 2023-12-12T02:05:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2023-12-12T02:05:04.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:04.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=129 2023-12-12T02:05:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:04.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:04.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-12T02:05:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:04.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:04.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=73 2023-12-12T02:05:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 129 = 73 2023-12-12T02:05:04.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-12T02:05:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:04.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 129 = 2023-12-12T02:05:04.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-12T02:05:05.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:05.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 129 = 2023-12-12T02:05:05.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:05:05.992 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.781273032687517) > weight (0.25) 2023-12-12T02:05:05.992 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5612109050763063) > weight (0.25) 2023-12-12T02:05:05.992 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-12T02:05:05.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-12T02:05:05.993 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T02:05:05.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:05:05.999+0000 7f03450ed640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 52419) UID: 1000 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.status 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2023-12-12T02:05:07.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:07.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=129 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 129 = 129 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:07.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:08.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2023-12-12T02:05:08.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_6': 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de25e8b1eb 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de25e8b1eb 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:03 2023 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:03 2023 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:03 2023 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:08.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0e8bda32-db29-49f2-b777-53035364d63f 2023-12-12T02:05:08.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_6 snap 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.export 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.export 2023-12-12T02:05:08.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.export 2023-12-12T02:05:08.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.export 2023-12-12T02:05:08.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:08.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.export 2023-12-12T02:05:08.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:09.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.export 2023-12-12T02:05:09.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_6.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_6.export 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_7 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_7 snap 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:09.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2023-12-12T02:05:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:10.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_7 2023-12-12T02:05:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:11.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:11.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2023-12-12T02:05:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:11.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:11.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:11.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:11.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2023-12-12T02:05:11.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2023-12-12T02:05:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:11.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:11.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:11.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:11.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:05:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:11.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2023-12-12T02:05:11.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:11.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:11.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:11.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_7.status 2023-12-12T02:05:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_7.status 2023-12-12T02:05:11.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.status 2023-12-12T02:05:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_7.status 2023-12-12T02:05:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_7.status 2023-12-12T02:05:11.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:11.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2023-12-12T02:05:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 131 2023-12-12T02:05:12.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.status 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.status 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2023-12-12T02:05:12.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:12.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=131 2023-12-12T02:05:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:12.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-12T02:05:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:12.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:12.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 131 = 2023-12-12T02:05:12.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-12T02:05:12.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:12.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:12.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 131 = 2023-12-12T02:05:12.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:12.993 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-12T02:05:12.993 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:05:12.994 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-12T02:05:12.994 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-12T02:05:12.994 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-12T02:05:12.997 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-12T02:05:12.997 INFO:tasks.rbd_mirror_thrash:waiting for 74 secs before thrashing 2023-12-12T02:05:13.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.status 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2023-12-12T02:05:13.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=131 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 131 = 131 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:13.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:14.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2023-12-12T02:05:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_7': 2023-12-12T02:05:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:14.907 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 125e13abc0dd 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.125e13abc0dd 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:12 2023 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:12 2023 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:12 2023 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: ce49be9c-8844-4083-8907-bd68b7a39229 2023-12-12T02:05:14.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:14.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_7 snap 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.export 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.export 2023-12-12T02:05:14.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.export 2023-12-12T02:05:14.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.export 2023-12-12T02:05:15.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:15.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.export 2023-12-12T02:05:15.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:15.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.export 2023-12-12T02:05:15.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_7.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_7.export 2023-12-12T02:05:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:15.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_8 2023-12-12T02:05:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_8 snap 2023-12-12T02:05:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:15.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:16.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:16.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:17.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:17.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_8 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:17.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2023-12-12T02:05:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:17.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:17.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:17.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2023-12-12T02:05:17.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:17.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_8.status 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_8.status 2023-12-12T02:05:17.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:17.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.status 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_8.status 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_8.status 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2023-12-12T02:05:18.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 133 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.status 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.status 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2023-12-12T02:05:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:18.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=133 2023-12-12T02:05:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:18.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-12T02:05:18.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 133 = 2023-12-12T02:05:18.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-12T02:05:18.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:18.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 133 = 2023-12-12T02:05:19.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.status 2023-12-12T02:05:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2023-12-12T02:05:19.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=133 2023-12-12T02:05:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 133 = 133 2023-12-12T02:05:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:19.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:20.533 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T02:05:20.528+0000 7f2a08a08640 -1 reset not still connected to 0x55c66b607a00 2023-12-12T02:05:20.533 INFO:tasks.ceph.cluster1.osd.2.smithi055.stderr:2023-12-12T02:05:20.528+0000 7f2a08a08640 -1 reset not still connected to 0x55c66c7c4340 2023-12-12T02:05:20.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_8': 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12961d028919 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12961d028919 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:27 2023 2023-12-12T02:05:20.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:27 2023 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:27 2023 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 342b1cab-88ee-458a-94d9-2ad59864a055 2023-12-12T02:05:20.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_8 snap 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.export 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.export 2023-12-12T02:05:20.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.export 2023-12-12T02:05:20.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.export 2023-12-12T02:05:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.export 2023-12-12T02:05:21.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:21.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.export 2023-12-12T02:05:21.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_8.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_8.export 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_9 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_9 snap 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:21.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2023-12-12T02:05:22.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:22.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2023-12-12T02:05:23.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:23.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_9 2023-12-12T02:05:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:23.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:23.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:23.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:23.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2023-12-12T02:05:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:23.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:23.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:23.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_9.status 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_9.status 2023-12-12T02:05:23.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.status 2023-12-12T02:05:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_9.status 2023-12-12T02:05:23.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_9.status 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:23.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2023-12-12T02:05:24.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 135 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.status 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.status 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2023-12-12T02:05:24.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:24.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=135 2023-12-12T02:05:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:24.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:24.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-12T02:05:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:24.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=76 2023-12-12T02:05:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 135 = 76 2023-12-12T02:05:24.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:24.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-12T02:05:24.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:25.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:25.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 135 = 2023-12-12T02:05:25.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:25.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-12T02:05:25.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:25.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 135 = 2023-12-12T02:05:25.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:05:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.status 2023-12-12T02:05:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2023-12-12T02:05:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:27.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=135 2023-12-12T02:05:27.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 135 = 135 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:27.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:28.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2023-12-12T02:05:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_9': 2023-12-12T02:05:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:28.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10dea2640714 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10dea2640714 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:37 2023 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:37 2023 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:37 2023 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 55c1b0cd-97fd-4de6-bd65-a62b4f88d578 2023-12-12T02:05:28.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:28.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_9 snap 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.export 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.export 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.export 2023-12-12T02:05:28.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.export 2023-12-12T02:05:28.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:28.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.export 2023-12-12T02:05:29.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:29.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.export 2023-12-12T02:05:29.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_9.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_9.export 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_10 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_10 snap 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:29.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:29.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:30.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:31.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:31.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_10 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:31.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:31.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:31.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:31.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:31.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:31.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2023-12-12T02:05:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:31.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:31.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_10.status 2023-12-12T02:05:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_10.status 2023-12-12T02:05:31.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:05:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.status 2023-12-12T02:05:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_10.status 2023-12-12T02:05:31.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_10.status 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:31.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2023-12-12T02:05:32.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 137 2023-12-12T02:05:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.status 2023-12-12T02:05:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:32.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.status 2023-12-12T02:05:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2023-12-12T02:05:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:32.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=137 2023-12-12T02:05:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:32.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-12T02:05:32.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:32.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:32.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 137 = 2023-12-12T02:05:32.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:32.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-12T02:05:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:32.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 137 = 2023-12-12T02:05:32.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:33.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.status 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2023-12-12T02:05:33.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:33.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:33.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=137 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 137 = 137 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:33.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:34.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2023-12-12T02:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_10': 2023-12-12T02:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:34.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b42d3791e 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b42d3791e 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:48 2023 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:48 2023 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:48 2023 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 73a5838f-6323-4afc-bd11-554e7ecf7c56 2023-12-12T02:05:34.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_10 snap 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.export 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.export 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.export 2023-12-12T02:05:34.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.export 2023-12-12T02:05:35.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:35.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.export 2023-12-12T02:05:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:35.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.export 2023-12-12T02:05:35.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_10.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_10.export 2023-12-12T02:05:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_11 2023-12-12T02:05:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_11 snap 2023-12-12T02:05:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:35.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:36.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:37.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_11 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:37.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:37.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:37.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:37.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2023-12-12T02:05:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-12T02:05:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:37.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2023-12-12T02:05:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:05:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:37.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:37.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:37.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:05:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2023-12-12T02:05:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2023-12-12T02:05:37.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:37.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:05:37.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:05:37.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:37.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:05:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2023-12-12T02:05:37.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T02:05:37.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T02:05:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:37.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:37.605 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_11.status 2023-12-12T02:05:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_11.status 2023-12-12T02:05:37.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.status 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_11.status 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_11.status 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:37.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2023-12-12T02:05:38.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 139 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.status 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.status 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2023-12-12T02:05:38.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:38.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=139 2023-12-12T02:05:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:38.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-12T02:05:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:38.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:38.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 139 = 2023-12-12T02:05:38.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-12T02:05:38.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:39.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:39.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 139 = 2023-12-12T02:05:39.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.status 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2023-12-12T02:05:39.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:39.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=139 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 139 = 139 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:39.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:39.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:40.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2023-12-12T02:05:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_11': 2023-12-12T02:05:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:40.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 125efd3959fa 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.125efd3959fa 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:56:56 2023 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:56:56 2023 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:56:56 2023 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: dfbf3fd7-eb15-4e16-a0ac-43c2a8e3e631 2023-12-12T02:05:40.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_11 snap 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.export 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.export 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.export 2023-12-12T02:05:40.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.export 2023-12-12T02:05:41.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:41.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.export 2023-12-12T02:05:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:41.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.export 2023-12-12T02:05:41.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_11.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_11.export 2023-12-12T02:05:41.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_12 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_12 snap 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:41.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2023-12-12T02:05:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:42.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2023-12-12T02:05:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:43.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:43.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i stateadmin_daemons 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: cluster1 rbd mirror status mirror/image_12 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:43.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:43.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:43.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2023-12-12T02:05:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2023-12-12T02:05:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:43.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:43.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:43.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:43.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:05:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:43.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2023-12-12T02:05:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:43.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2023-12-12T02:05:43.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_12.status 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_12.status 2023-12-12T02:05:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.status 2023-12-12T02:05:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_12.status 2023-12-12T02:05:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_12.status 2023-12-12T02:05:43.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:43.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2023-12-12T02:05:44.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 141 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.status 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.status 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2023-12-12T02:05:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:44.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:44.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=141 2023-12-12T02:05:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:44.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-12T02:05:44.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:44.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=79 2023-12-12T02:05:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 141 = 79 2023-12-12T02:05:44.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:44.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-12T02:05:44.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:45.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 141 = 2023-12-12T02:05:45.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.status 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2023-12-12T02:05:45.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:45.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=141 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 141 = 141 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:45.905 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:46.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2023-12-12T02:05:46.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_12': 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120ba7513cd 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120ba7513cd 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:32 2023 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:46.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: ddfde94a-a349-4e2a-ae4d-72258f5591a6 2023-12-12T02:05:46.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_12 snap 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.export 2023-12-12T02:05:46.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.export 2023-12-12T02:05:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.export 2023-12-12T02:05:46.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.export 2023-12-12T02:05:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:47.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.export 2023-12-12T02:05:47.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:47.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.export 2023-12-12T02:05:47.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_12.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_12.export 2023-12-12T02:05:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_13 2023-12-12T02:05:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_13 snap 2023-12-12T02:05:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:47.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:47.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:47.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:47.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2023-12-12T02:05:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:48.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror statusgrep mirror/image_13 -i state 2023-12-12T02:05:48.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2023-12-12T02:05:48.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:48.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:48.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:48.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2023-12-12T02:05:49.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo "state": "Replaying" 2023-12-12T02:05:49.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T02:05:49.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:49.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_13.status 2023-12-12T02:05:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_13.status 2023-12-12T02:05:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.status 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_13.status 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_13.status 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:49.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:49.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2023-12-12T02:05:49.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 143 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.status 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.status 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2023-12-12T02:05:49.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:49.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=143 2023-12-12T02:05:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:50.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-12T02:05:50.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=80 2023-12-12T02:05:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 143 = 80 2023-12-12T02:05:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-12T02:05:50.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:50.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 143 = 2023-12-12T02:05:50.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-12T02:05:51.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:51.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:51.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 143 = 2023-12-12T02:05:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:05:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:53.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.status 2023-12-12T02:05:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2023-12-12T02:05:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:53.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=143 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 143 = 143 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:53.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:54.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2023-12-12T02:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_13': 2023-12-12T02:05:54.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129ee6bb8bca 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129ee6bb8bca 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:12 2023 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:12 2023 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:12 2023 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:54.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 6c841662-f2e7-4c7e-8c09-62dd32e1eff8 2023-12-12T02:05:54.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_13 snap 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.export 2023-12-12T02:05:54.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.export 2023-12-12T02:05:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.export 2023-12-12T02:05:54.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.export 2023-12-12T02:05:54.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:54.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.export 2023-12-12T02:05:54.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:05:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.export 2023-12-12T02:05:54.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_13.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_13.export 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_14 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_14 snap 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:55.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:55.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:05:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:05:56.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/image_14 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i statelocal 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: cluster_instance=cluster1 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:05:56.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:05:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:05:56.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:56.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:05:56.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2023-12-12T02:05:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-12T02:05:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:56.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2023-12-12T02:05:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:05:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:56.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:05:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:05:56.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:05:56.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:05:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:05:56.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2023-12-12T02:05:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2023-12-12T02:05:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:05:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:56.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:05:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:05:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:05:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:05:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:05:56.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:05:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:56.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:05:56.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2023-12-12T02:05:56.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:05:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:05:56.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:05:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:05:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_14.status 2023-12-12T02:05:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_14.status 2023-12-12T02:05:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.status 2023-12-12T02:05:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_14.status 2023-12-12T02:05:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_14.status 2023-12-12T02:05:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2023-12-12T02:05:57.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 145 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.status 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.status 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2023-12-12T02:05:57.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:57.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:05:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=145 2023-12-12T02:05:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:57.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:57.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-12T02:05:57.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:57.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 145 = 2023-12-12T02:05:57.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-12T02:05:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:05:58.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 145 = 2023-12-12T02:05:58.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:05:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:58.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.status 2023-12-12T02:05:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2023-12-12T02:05:58.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:05:58.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:05:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=145 2023-12-12T02:05:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 145 = 145 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:05:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:05:59.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2023-12-12T02:05:59.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_14': 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e5a70f0ab 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e5a70f0ab 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:26 2023 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:26 2023 2023-12-12T02:05:59.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:26 2023 2023-12-12T02:05:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:05:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:05:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:05:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c9956d0a-2c28-4015-9421-074ee875e996 2023-12-12T02:05:59.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_14 snap 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.export 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.export 2023-12-12T02:05:59.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.export 2023-12-12T02:05:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.export 2023-12-12T02:06:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.export 2023-12-12T02:06:00.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:00.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.export 2023-12-12T02:06:00.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_14.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_14.export 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_15 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_15 snap 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:00.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:00.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2023-12-12T02:06:01.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:01.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:02.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_15 2023-12-12T02:06:02.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1grep 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i+ statelocal 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: loop_instance 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:02.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:02.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:02.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:02.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2023-12-12T02:06:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-12T02:06:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:02.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:02.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:02.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:02.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2023-12-12T02:06:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-12T02:06:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:03.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2023-12-12T02:06:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:06:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:06:03.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:03.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:03.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2023-12-12T02:06:03.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2023-12-12T02:06:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_15 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:03.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:06:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:03.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:06:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:03.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_15 2023-12-12T02:06:03.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:03.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:03.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:03.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:03.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_15.status 2023-12-12T02:06:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_15.status 2023-12-12T02:06:03.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.status 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_15.status 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_15.status 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:03.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2023-12-12T02:06:03.779 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 147 2023-12-12T02:06:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.status 2023-12-12T02:06:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:03.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.status 2023-12-12T02:06:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2023-12-12T02:06:03.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:03.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=147 2023-12-12T02:06:03.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:03.848 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-12T02:06:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:04.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=82 2023-12-12T02:06:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 147 = 82 2023-12-12T02:06:04.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-12T02:06:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:04.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:04.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 147 = 2023-12-12T02:06:04.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-12T02:06:05.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 147 = 2023-12-12T02:06:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.status 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2023-12-12T02:06:07.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:07.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=147 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 147 = 147 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:07.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:08.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_15': 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 125e3cccc93c 2023-12-12T02:06:08.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.125e3cccc93c 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:35 2023 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:35 2023 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:35 2023 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 09d8c9b1-86e1-481e-8a12-992f2d80f06d 2023-12-12T02:06:08.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_15 snap 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.export 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.export 2023-12-12T02:06:08.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.export 2023-12-12T02:06:08.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.export 2023-12-12T02:06:08.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:08.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.export 2023-12-12T02:06:09.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:09.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.export 2023-12-12T02:06:09.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_15.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_15.export 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_16 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_16 snap 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2023-12-12T02:06:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:10.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2023-12-12T02:06:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:11.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/image_16 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:local cluster_instance=cluster1 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:11.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2023-12-12T02:06:11.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:11.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:11.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:11.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2023-12-12T02:06:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2023-12-12T02:06:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:11.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2023-12-12T02:06:11.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:11.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:11.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:11.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2023-12-12T02:06:11.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo "state": "Replaying" 2023-12-12T02:06:11.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T02:06:11.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:11.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_16.status 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_16.status 2023-12-12T02:06:11.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.status 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_16.status 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_16.status 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2023-12-12T02:06:11.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:11.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2023-12-12T02:06:12.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 149 2023-12-12T02:06:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.status 2023-12-12T02:06:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:12.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.status 2023-12-12T02:06:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2023-12-12T02:06:12.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:12.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=149 2023-12-12T02:06:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-12T02:06:12.454 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:12.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:12.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 149 = 2023-12-12T02:06:12.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:12.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-12T02:06:12.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:13.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 149 = 2023-12-12T02:06:13.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.status 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2023-12-12T02:06:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:13.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=149 2023-12-12T02:06:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 149 = 149 2023-12-12T02:06:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:13.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2023-12-12T02:06:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_16': 2023-12-12T02:06:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:14.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e74717db2 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e74717db2 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:46 2023 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:46 2023 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:46 2023 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:14.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: cf75936b-449b-47ab-8ea6-3647877a3e66 2023-12-12T02:06:14.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_16 snap 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.export 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.export 2023-12-12T02:06:14.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.export 2023-12-12T02:06:14.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.export 2023-12-12T02:06:15.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:15.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.export 2023-12-12T02:06:15.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:15.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.export 2023-12-12T02:06:15.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_16.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_16.export 2023-12-12T02:06:15.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_17 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_17 snap 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:15.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:16.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:17.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_17 2023-12-12T02:06:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:06:17.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:17.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:17.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:17.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2023-12-12T02:06:17.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2023-12-12T02:06:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:17.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:17.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:17.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:17.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:17.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:17.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2023-12-12T02:06:17.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T02:06:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T02:06:17.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_17.status 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_17.status 2023-12-12T02:06:17.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.status 2023-12-12T02:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_17.status 2023-12-12T02:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_17.status 2023-12-12T02:06:17.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:17.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2023-12-12T02:06:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 151 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.status 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.status 2023-12-12T02:06:18.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2023-12-12T02:06:18.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:18.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=151 2023-12-12T02:06:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-12T02:06:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:18.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:18.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 151 = 2023-12-12T02:06:18.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-12T02:06:18.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:19.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 151 = 2023-12-12T02:06:19.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:19.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.status 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2023-12-12T02:06:19.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:19.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:19.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=151 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 151 = 151 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:20.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_17': 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de597933a8 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de597933a8 2023-12-12T02:06:20.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:57:57 2023 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:57:57 2023 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:57:57 2023 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d06e3eb6-22a3-4ffe-88ca-04ba98f63457 2023-12-12T02:06:20.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:20.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_17 snap 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.export 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.export 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.export 2023-12-12T02:06:20.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.export 2023-12-12T02:06:21.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:21.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.export 2023-12-12T02:06:21.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.export 2023-12-12T02:06:21.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_17.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_17.export 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_18 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_18 snap 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2023-12-12T02:06:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:22.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:22.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:23.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:23.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_18 2023-12-12T02:06:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:06:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local instance=cluster1 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i statesleep 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 0 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:23.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:23.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:23.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:23.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:23.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:23.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2023-12-12T02:06:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:23.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:23.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:23.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_18.status 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_18.statussed 2023-12-12T02:06:23.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -e s|[/ ]|_|g 2023-12-12T02:06:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.status 2023-12-12T02:06:23.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_18.status 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_18.status 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:23.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2023-12-12T02:06:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 153 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.status 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.status 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2023-12-12T02:06:24.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:24.299 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=153 2023-12-12T02:06:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:24.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-12T02:06:24.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:24.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=85 2023-12-12T02:06:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 153 = 85 2023-12-12T02:06:24.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:24.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-12T02:06:24.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:25.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:25.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 153 = 2023-12-12T02:06:25.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.status 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2023-12-12T02:06:25.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=153 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 153 = 153 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:25.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:25.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:26.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_18': 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b250db268 2023-12-12T02:06:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b250db268 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:11 2023 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:11 2023 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:11 2023 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 2db15675-ec89-40f1-b944-460662d253f6 2023-12-12T02:06:26.971 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_18 snap 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.export 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.export 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.export 2023-12-12T02:06:26.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.export 2023-12-12T02:06:26.998 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.8438576059906338) > weight (0.25) 2023-12-12T02:06:26.998 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4788903181630595) > weight (0.25) 2023-12-12T02:06:26.998 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.653678672551984) > weight (0.25) 2023-12-12T02:06:26.998 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5776394851204174) > weight (0.25) 2023-12-12T02:06:26.998 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-12T02:06:27.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:27.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.export 2023-12-12T02:06:27.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:27.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.export 2023-12-12T02:06:27.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_18.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_18.export 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_19 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_19 snap 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:27.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:27.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:27.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2023-12-12T02:06:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_19 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local loop_instance 2023-12-12T02:06:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:28.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2023-12-12T02:06:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-12T02:06:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2023-12-12T02:06:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:28.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:28.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:28.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2023-12-12T02:06:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2023-12-12T02:06:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_19 2023-12-12T02:06:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:06:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:06:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:06:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:29.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_19 2023-12-12T02:06:29.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:29.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:29.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_19.status 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_19.status 2023-12-12T02:06:29.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.status 2023-12-12T02:06:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_19.status 2023-12-12T02:06:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_19.status 2023-12-12T02:06:29.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:29.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2023-12-12T02:06:29.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 155 2023-12-12T02:06:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.status 2023-12-12T02:06:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:29.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.status 2023-12-12T02:06:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2023-12-12T02:06:29.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:29.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=155 2023-12-12T02:06:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:30.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-12T02:06:30.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:30.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=86 2023-12-12T02:06:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 155 = 86 2023-12-12T02:06:30.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:30.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-12T02:06:30.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:30.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=86 2023-12-12T02:06:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 155 = 86 2023-12-12T02:06:30.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-12T02:06:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:31.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:31.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 155 = 2023-12-12T02:06:31.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:06:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:33.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.status 2023-12-12T02:06:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2023-12-12T02:06:33.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=155 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 155 = 155 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:33.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:34.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8282428564806417) > weight (0.25) 2023-12-12T02:06:34.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9112602844660658) > weight (0.25) 2023-12-12T02:06:34.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.73221617176558) > weight (0.25) 2023-12-12T02:06:34.003 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.36683770017887984) > weight (0.25) 2023-12-12T02:06:34.004 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-12T02:06:34.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2023-12-12T02:06:34.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_19': 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 125ed7e79df8 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.125ed7e79df8 2023-12-12T02:06:34.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:21 2023 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:21 2023 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:21 2023 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 963f10a2-9317-4e5e-b35a-fc335be2a14a 2023-12-12T02:06:34.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_19 snap 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:34.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.export 2023-12-12T02:06:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.export 2023-12-12T02:06:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.export 2023-12-12T02:06:34.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.export 2023-12-12T02:06:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:34.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.export 2023-12-12T02:06:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.export 2023-12-12T02:06:35.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_19.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_19.export 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_20 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_20 snap 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:36.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2023-12-12T02:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:37.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:37.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_20 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shiftgrep 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:37.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2023-12-12T02:06:37.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:37.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:37.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:37.184 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:37.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2023-12-12T02:06:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2023-12-12T02:06:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:37.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:37.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:37.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2023-12-12T02:06:37.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:37.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_20.status 2023-12-12T02:06:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_20.status 2023-12-12T02:06:37.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.status 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_20.status 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_20.status 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2023-12-12T02:06:38.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 157 2023-12-12T02:06:38.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.status 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.status 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2023-12-12T02:06:38.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=157 2023-12-12T02:06:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:38.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:38.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-12T02:06:38.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:38.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=87 2023-12-12T02:06:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 157 = 87 2023-12-12T02:06:38.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-12T02:06:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:39.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 157 = 2023-12-12T02:06:39.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:39.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-12T02:06:39.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:39.843 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:06:39.845+0000 7fd7ab621640 -1 reset not still connected to 0x5574981fdc70 2023-12-12T02:06:39.843 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:06:39.845+0000 7fd7ab621640 -1 reset not still connected to 0x5574a2f26b60 2023-12-12T02:06:39.843 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:06:39.845+0000 7fd7ab621640 -1 reset not still connected to 0x5574a7a4e410 2023-12-12T02:06:39.844 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:06:39.845+0000 7fd7ab621640 -1 reset not still connected to 0x5574b0969790 2023-12-12T02:06:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:39.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 157 = 2023-12-12T02:06:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:06:41.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.status 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2023-12-12T02:06:41.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:41.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=157 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 157 = 157 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:41.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:42.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_20': 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e38ebf75 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e38ebf75 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:31 2023 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:31 2023 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:31 2023 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 74a54c46-73ec-4d02-aaa0-86f3e226c449 2023-12-12T02:06:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_20 snap 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.export 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.export 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.export 2023-12-12T02:06:42.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.export 2023-12-12T02:06:42.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:42.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.export 2023-12-12T02:06:43.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:43.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.export 2023-12-12T02:06:43.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_20.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_20.export 2023-12-12T02:06:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_21 2023-12-12T02:06:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_21 snap 2023-12-12T02:06:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:44.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:45.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_21 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep+ local -i cluster=cluster1 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:45.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:45.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:45.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2023-12-12T02:06:45.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:45.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:45.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_21.status 2023-12-12T02:06:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_21.status 2023-12-12T02:06:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.status 2023-12-12T02:06:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_21.status 2023-12-12T02:06:45.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_21.status 2023-12-12T02:06:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:45.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2023-12-12T02:06:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:45.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2023-12-12T02:06:46.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 159 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.status 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.status 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2023-12-12T02:06:46.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:46.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=159 2023-12-12T02:06:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:46.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-12T02:06:46.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:46.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=88 2023-12-12T02:06:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 159 = 88 2023-12-12T02:06:46.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-12T02:06:46.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:47.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 159 = 2023-12-12T02:06:47.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:47.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.status 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2023-12-12T02:06:47.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:47.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=159 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 159 = 159 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:47.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_21': 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a860aa8558 2023-12-12T02:06:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a860aa8558 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:43 2023 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:43 2023 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:43 2023 2023-12-12T02:06:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 59ede7a8-6346-4103-a0a4-f50800c189ac 2023-12-12T02:06:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_21 snap 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:48.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.export 2023-12-12T02:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.export 2023-12-12T02:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.export 2023-12-12T02:06:48.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.export 2023-12-12T02:06:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:49.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.export 2023-12-12T02:06:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:49.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.export 2023-12-12T02:06:49.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_21.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_21.export 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_22 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_22 snap 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:50.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:50.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:51.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:51.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_22 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:51.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:51.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:51.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:51.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:51.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2023-12-12T02:06:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-12T02:06:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:51.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:51.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:51.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:51.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2023-12-12T02:06:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2023-12-12T02:06:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:51.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2023-12-12T02:06:51.501 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:06:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:06:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:51.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2023-12-12T02:06:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:06:51.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:06:51.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:51.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_22.status 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_22.status 2023-12-12T02:06:51.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.status 2023-12-12T02:06:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_22.status 2023-12-12T02:06:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_22.status 2023-12-12T02:06:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:51.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2023-12-12T02:06:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:51.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2023-12-12T02:06:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 161 2023-12-12T02:06:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.status 2023-12-12T02:06:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.status 2023-12-12T02:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2023-12-12T02:06:52.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:52.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=161 2023-12-12T02:06:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:06:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:52.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-12T02:06:52.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:52.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=89 2023-12-12T02:06:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 161 = 89 2023-12-12T02:06:52.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-12T02:06:52.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:53.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 161 = 2023-12-12T02:06:53.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:53.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-12T02:06:53.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:53.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 161 = 2023-12-12T02:06:53.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:06:55.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.status 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2023-12-12T02:06:55.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=161 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 161 = 161 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:55.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:56.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2023-12-12T02:06:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_22': 2023-12-12T02:06:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:06:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:06:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:06:56.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a83c84160c 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a83c84160c 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:58:54 2023 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:58:54 2023 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:58:54 2023 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: e51faaaf-277b-490b-9e12-ad199fdee9c2 2023-12-12T02:06:56.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_22 snap 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.export 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.export 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.export 2023-12-12T02:06:56.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.export 2023-12-12T02:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.export 2023-12-12T02:06:57.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:06:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.export 2023-12-12T02:06:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_22.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_22.export 2023-12-12T02:06:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:57.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_23 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_23 snap 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:06:57.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:06:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2023-12-12T02:06:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:57.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:06:57.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:06:58.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_23 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1+ 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ greplocal -i instance=cluster1 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: state 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:06:58.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:06:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:06:58.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:06:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2023-12-12T02:06:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:06:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:06:58.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:06:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:58.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:06:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2023-12-12T02:06:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-12T02:06:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:06:58.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:06:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:06:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2023-12-12T02:06:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-12T02:06:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:59.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:59.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:06:59.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:06:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:06:59.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2023-12-12T02:06:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2023-12-12T02:06:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:06:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:06:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2023-12-12T02:06:59.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:06:59.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:06:59.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:06:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:06:59.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2023-12-12T02:06:59.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:06:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo+ "state": "Replaying" 2023-12-12T02:06:59.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T02:06:59.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:06:59.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_23.status 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_23.status 2023-12-12T02:06:59.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.status 2023-12-12T02:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_23.status 2023-12-12T02:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_23.status 2023-12-12T02:06:59.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:59.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2023-12-12T02:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 163 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.status 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.status 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2023-12-12T02:06:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:06:59.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:06:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=163 2023-12-12T02:06:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:06:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:00.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-12T02:07:00.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:00.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=90 2023-12-12T02:07:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 163 = 90 2023-12-12T02:07:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-12T02:07:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=90 2023-12-12T02:07:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 163 = 90 2023-12-12T02:07:00.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-12T02:07:01.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:01.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:01.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 163 = 2023-12-12T02:07:01.499 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:07:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:03.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.status 2023-12-12T02:07:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2023-12-12T02:07:03.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:03.173 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:03.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=163 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 163 = 163 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:04.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_23': 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129ee9793d69 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129ee9793d69 2023-12-12T02:07:04.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:03 2023 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:03 2023 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:03 2023 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a4a6e595-cc0c-4cac-b030-d274e817dada 2023-12-12T02:07:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:04.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_23 snap 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.export 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.export 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.export 2023-12-12T02:07:04.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.export 2023-12-12T02:07:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:04.465 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.export 2023-12-12T02:07:04.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:04.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.export 2023-12-12T02:07:04.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_23.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_23.export 2023-12-12T02:07:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_24 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_24 snap 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:04.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2023-12-12T02:07:05.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:05.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2023-12-12T02:07:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:06.187 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_24 2023-12-12T02:07:06.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + sleep 0 2023-12-12T02:07:06.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:07:06.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:06.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:06.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:06.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2023-12-12T02:07:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:06.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:06.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2023-12-12T02:07:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2023-12-12T02:07:06.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:06.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2023-12-12T02:07:07.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 165 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.status 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.status 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2023-12-12T02:07:07.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:07.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=165 2023-12-12T02:07:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:07.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:07.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-12T02:07:07.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:07.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 165 = 2023-12-12T02:07:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:07.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-12T02:07:07.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:08.047 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:08.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 165 = 2023-12-12T02:07:08.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:08.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.status 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2023-12-12T02:07:08.852 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:08.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=165 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 165 = 165 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:08.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:09.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_24': 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a8aa4681da 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a8aa4681da 2023-12-12T02:07:09.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:12 2023 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:12 2023 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:12 2023 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d213a902-dc35-4507-886a-3f4b1129d6d2 2023-12-12T02:07:09.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_24 snap 2023-12-12T02:07:09.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.export 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.export 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.export 2023-12-12T02:07:09.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.export 2023-12-12T02:07:10.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.export 2023-12-12T02:07:10.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:10.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.export 2023-12-12T02:07:10.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_24.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_24.export 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_25 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_25 snap 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:11.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:11.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_25 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:12.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:12.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:07:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:12.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:12.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:12.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2023-12-12T02:07:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_25'] 2023-12-12T02:07:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:12.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_25 2023-12-12T02:07:12.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:12.378 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:12.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:12.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:12.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_25 2023-12-12T02:07:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:12.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo "state": "Replaying" 2023-12-12T02:07:12.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T02:07:12.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:12.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_25.status 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_25.status 2023-12-12T02:07:12.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.status 2023-12-12T02:07:12.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_25.status 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_25.status 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:12.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2023-12-12T02:07:13.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 167 2023-12-12T02:07:13.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.status 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.status 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2023-12-12T02:07:13.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=167 2023-12-12T02:07:13.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:13.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:13.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-12T02:07:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:13.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=92 2023-12-12T02:07:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 167 = 92 2023-12-12T02:07:13.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:14.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-12T02:07:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:14.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:14.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 167 = 2023-12-12T02:07:14.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:14.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.status 2023-12-12T02:07:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:14.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2023-12-12T02:07:14.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:14.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=167 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 167 = 167 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:14.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:15.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2023-12-12T02:07:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_25': 2023-12-12T02:07:16.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de1a404936 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de1a404936 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:18 2023 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:18 2023 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:18 2023 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:16.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a921f597-6426-4126-a233-e3c16f3533b3 2023-12-12T02:07:16.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_25 snap 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.export 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.export 2023-12-12T02:07:16.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.export 2023-12-12T02:07:16.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.export 2023-12-12T02:07:16.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:16.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.export 2023-12-12T02:07:16.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.export 2023-12-12T02:07:16.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_25.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_25.export 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_26 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_26 snap 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:16.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:16.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:16.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:16.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:17.778 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_26 2023-12-12T02:07:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:07:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:17.782 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:17.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:17.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:17.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:17.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:17.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2023-12-12T02:07:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2023-12-12T02:07:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:17.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2023-12-12T02:07:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:17.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:17.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:17.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:17.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:17.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:17.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:17.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:17.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2023-12-12T02:07:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:18.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:18.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:18.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:18.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_26.status 2023-12-12T02:07:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_26.status 2023-12-12T02:07:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.status 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_26.status 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_26.status 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:18.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2023-12-12T02:07:18.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 169 2023-12-12T02:07:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.status 2023-12-12T02:07:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:18.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.status 2023-12-12T02:07:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2023-12-12T02:07:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=169 2023-12-12T02:07:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:19.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-12T02:07:19.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:19.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=93 2023-12-12T02:07:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 169 = 93 2023-12-12T02:07:19.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-12T02:07:19.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:19.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:19.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:19.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 169 = 2023-12-12T02:07:19.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-12T02:07:20.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:20.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:20.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 169 = 2023-12-12T02:07:20.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:07:22.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.status 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2023-12-12T02:07:22.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=169 2023-12-12T02:07:22.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 169 = 169 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:23.166 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_26': 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b1c347f53 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b1c347f53 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:23.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:32 2023 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:32 2023 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:32 2023 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 26f2fa83-50d9-4f27-b56d-3181fdc135a5 2023-12-12T02:07:23.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_26 snap 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.export 2023-12-12T02:07:23.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.export 2023-12-12T02:07:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.export 2023-12-12T02:07:23.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.export 2023-12-12T02:07:23.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.export 2023-12-12T02:07:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:23.770 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.export 2023-12-12T02:07:23.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_26.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_26.export 2023-12-12T02:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:23.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_27 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_27 snap 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:23.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:24.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:24.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:25.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_27 2023-12-12T02:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:25.201 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:25.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:25.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:25.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:25.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:25.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:25.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2023-12-12T02:07:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2023-12-12T02:07:25.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:25.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:25.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:25.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:25.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:25.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:25.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:25.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2023-12-12T02:07:25.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:25.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:25.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:25.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_27.status 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_27.status 2023-12-12T02:07:25.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.status 2023-12-12T02:07:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_27.status 2023-12-12T02:07:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster1-mirror-image_27.status 2023-12-12T02:07:25.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:25.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2023-12-12T02:07:26.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 171 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.status 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.status 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2023-12-12T02:07:26.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:26.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=171 2023-12-12T02:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:26.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-12T02:07:26.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:26.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=94 2023-12-12T02:07:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 171 = 94 2023-12-12T02:07:26.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-12T02:07:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 171 = 2023-12-12T02:07:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.status 2023-12-12T02:07:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2023-12-12T02:07:27.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:27.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=171 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 171 = 171 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:27.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:27.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2023-12-12T02:07:28.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_27': 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e9725167d 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e9725167d 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:45 2023 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:45 2023 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:45 2023 2023-12-12T02:07:28.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 82b98cb4-5a2a-4924-8f4e-9681fc9a7a67 2023-12-12T02:07:28.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_27 snap 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.export 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.export 2023-12-12T02:07:28.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.export 2023-12-12T02:07:28.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.export 2023-12-12T02:07:29.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:29.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.export 2023-12-12T02:07:29.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:29.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.export 2023-12-12T02:07:29.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_27.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_27.export 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_28 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_28 snap 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:30.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:30.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:31.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + admin_daemons cluster1 rbd mirror status mirror/image_28 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep+ -i statelocal 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: cluster_instance=cluster1 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:31.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:31.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2023-12-12T02:07:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:31.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:31.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:31.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:31.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:31.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2023-12-12T02:07:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-12T02:07:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:31.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:31.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:31.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2023-12-12T02:07:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2023-12-12T02:07:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:07:31.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:07:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:07:31.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:31.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2023-12-12T02:07:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2023-12-12T02:07:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_28.status 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_28.status 2023-12-12T02:07:31.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.status 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_28.status 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_28.status 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2023-12-12T02:07:31.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2023-12-12T02:07:32.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 173 2023-12-12T02:07:32.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.status 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.status 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2023-12-12T02:07:32.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=173 2023-12-12T02:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:32.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-12T02:07:32.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=95 2023-12-12T02:07:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 173 = 95 2023-12-12T02:07:32.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:32.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-12T02:07:32.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:33.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 173 = 2023-12-12T02:07:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.status 2023-12-12T02:07:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2023-12-12T02:07:33.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=173 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 173 = 173 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:34.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_28': 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a8bd60a89e 2023-12-12T02:07:34.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a8bd60a89e 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 01:59:57 2023 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 01:59:57 2023 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 01:59:57 2023 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c9a7b9aa-ebf2-4ce9-8f2a-36162c41d7cf 2023-12-12T02:07:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_28 snap 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.export 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.export 2023-12-12T02:07:35.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.export 2023-12-12T02:07:35.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.export 2023-12-12T02:07:35.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:35.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.export 2023-12-12T02:07:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:35.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.export 2023-12-12T02:07:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_28.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_28.export 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_29 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_29 snap 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:35.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:36.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:36.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2023-12-12T02:07:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_29 2023-12-12T02:07:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:37.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:37.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2023-12-12T02:07:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-12T02:07:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:37.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2023-12-12T02:07:37.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:37.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:37.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:37.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2023-12-12T02:07:38.076 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-12T02:07:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_29 2023-12-12T02:07:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:07:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:38.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:38.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:38.081 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:07:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:38.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_29 2023-12-12T02:07:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2023-12-12T02:07:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:38.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:38.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_29 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:07:38.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:07:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:38.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_29 2023-12-12T02:07:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:38.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo "state": "Replaying" 2023-12-12T02:07:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i Replaying 2023-12-12T02:07:38.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:38.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:38.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_29.status 2023-12-12T02:07:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_29.status 2023-12-12T02:07:38.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.status 2023-12-12T02:07:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_29.status 2023-12-12T02:07:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_29.status 2023-12-12T02:07:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:38.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2023-12-12T02:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:38.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2023-12-12T02:07:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 175 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.status 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.status 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2023-12-12T02:07:38.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:38.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:38.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=175 2023-12-12T02:07:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:38.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:39.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-12T02:07:39.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:39.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=96 2023-12-12T02:07:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 175 = 96 2023-12-12T02:07:39.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-12T02:07:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:39.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=96 2023-12-12T02:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 175 = 96 2023-12-12T02:07:39.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-12T02:07:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:40.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 175 = 2023-12-12T02:07:40.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:07:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:42.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.status 2023-12-12T02:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2023-12-12T02:07:42.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:42.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:42.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=175 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 175 = 175 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:42.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:43.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_29': 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de5f368b01 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de5f368b01 2023-12-12T02:07:43.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:08 2023 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:08 2023 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:08 2023 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: d3da7272-827d-4801-89f4-6e6b377cdca7 2023-12-12T02:07:43.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_29 snap 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.export 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.export 2023-12-12T02:07:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.export 2023-12-12T02:07:43.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.export 2023-12-12T02:07:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:43.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.export 2023-12-12T02:07:43.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.export 2023-12-12T02:07:43.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_29.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_29.export 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_30 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_30 snap 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:43.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:44.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:44.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2023-12-12T02:07:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:45.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:45.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_30 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instancegrep 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:45.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:45.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:45.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:45.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:45.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:45.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2023-12-12T02:07:45.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-12T02:07:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:45.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2023-12-12T02:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:45.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:45.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:45.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:45.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:45.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2023-12-12T02:07:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2023-12-12T02:07:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:45.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:45.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:45.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:45.493 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:07:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:45.515 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:45.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2023-12-12T02:07:45.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:45.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:45.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_30.status 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_30.status 2023-12-12T02:07:45.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.status 2023-12-12T02:07:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_30.status 2023-12-12T02:07:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_30.status 2023-12-12T02:07:45.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:45.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2023-12-12T02:07:46.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 177 2023-12-12T02:07:46.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.status 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.status 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2023-12-12T02:07:46.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:46.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=177 2023-12-12T02:07:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:46.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-12T02:07:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:46.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=97 2023-12-12T02:07:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 177 = 97 2023-12-12T02:07:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-12T02:07:46.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:47.005 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T02:07:47.005 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T02:07:47.005 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T02:07:47.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:07:47.012+0000 7f242cc99640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 53215) UID: 1000 2023-12-12T02:07:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:47.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:47.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 177 = 2023-12-12T02:07:47.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.status 2023-12-12T02:07:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2023-12-12T02:07:47.869 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=177 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 177 = 177 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:48.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_30': 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b6ea3945 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b6ea3945 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:48.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:20 2023 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:20 2023 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:20 2023 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 37a61d89-b375-4db8-b09d-d9947b095e5c 2023-12-12T02:07:48.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_30 snap 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.export 2023-12-12T02:07:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.export 2023-12-12T02:07:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.export 2023-12-12T02:07:49.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.export 2023-12-12T02:07:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:49.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.export 2023-12-12T02:07:49.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:49.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.export 2023-12-12T02:07:49.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_30.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_30.export 2023-12-12T02:07:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_31 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_31 snap 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:49.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:50.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:50.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_31 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:07:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ greplocal -i loop_instance state 2023-12-12T02:07:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:51.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:51.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2023-12-12T02:07:51.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2023-12-12T02:07:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2023-12-12T02:07:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:51.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:51.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:51.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:51.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2023-12-12T02:07:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-12T02:07:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:51.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2023-12-12T02:07:51.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:51.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:51.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:07:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:07:51.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:07:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:51.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:07:51.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2023-12-12T02:07:51.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2023-12-12T02:07:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:51.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:51.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:07:51.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:07:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:07:51.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:07:51.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:07:51.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2023-12-12T02:07:51.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T02:07:51.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T02:07:51.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:51.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_31.status 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_31.status 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:07:51.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.status 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_31.status 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_31.status 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2023-12-12T02:07:51.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:51.770 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2023-12-12T02:07:52.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 179 2023-12-12T02:07:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.status 2023-12-12T02:07:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:52.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.status 2023-12-12T02:07:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2023-12-12T02:07:52.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:52.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:52.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=179 2023-12-12T02:07:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:52.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-12T02:07:52.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:52.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=98 2023-12-12T02:07:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 179 = 98 2023-12-12T02:07:52.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:07:52.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-12T02:07:52.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:53.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 179 = 2023-12-12T02:07:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-12T02:07:53.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:07:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:07:53.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 179 = 2023-12-12T02:07:53.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:07:54.006 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T02:07:54.007 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:07:54.007 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T02:07:54.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T02:07:54.007 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T02:07:54.010 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T02:07:54.010 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2023-12-12T02:07:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.status 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2023-12-12T02:07:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:55.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:07:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=179 2023-12-12T02:07:55.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 179 = 179 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:55.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2023-12-12T02:07:56.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_31': 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129edd9f883a 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129edd9f883a 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:32 2023 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:32 2023 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:32 2023 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:07:56.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:07:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: f5bbdd5f-8443-4677-b3b4-d498845c7bef 2023-12-12T02:07:56.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_31 snap 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.export 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.export 2023-12-12T02:07:56.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.export 2023-12-12T02:07:56.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.export 2023-12-12T02:07:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:56.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.export 2023-12-12T02:07:57.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:07:57.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.export 2023-12-12T02:07:57.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_31.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_31.export 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_32 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_32 snap 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:07:57.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:07:57.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:07:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:07:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_32 2023-12-12T02:07:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:07:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:07:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:07:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:07:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:07:58.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:07:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:07:58.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2023-12-12T02:07:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2023-12-12T02:07:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:07:58.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:07:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:07:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:07:58.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2023-12-12T02:07:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:07:59.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:07:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:07:59.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:07:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:07:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:07:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:07:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2023-12-12T02:07:59.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_32.status 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_32.status 2023-12-12T02:07:59.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.status 2023-12-12T02:07:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_32.status 2023-12-12T02:07:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_32.status 2023-12-12T02:07:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:07:59.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:07:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2023-12-12T02:07:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:59.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2023-12-12T02:07:59.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 181 2023-12-12T02:07:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.status 2023-12-12T02:07:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:07:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:07:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:07:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.status 2023-12-12T02:07:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2023-12-12T02:07:59.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:07:59.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:07:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=181 2023-12-12T02:07:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:07:59.934 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:00.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-12T02:08:00.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:00.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=99 2023-12-12T02:08:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 181 = 99 2023-12-12T02:08:00.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-12T02:08:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 181 = 2023-12-12T02:08:00.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:01.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-12T02:08:01.479 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:01.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 181 = 2023-12-12T02:08:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.status 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2023-12-12T02:08:03.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:03.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=181 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 181 = 181 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:03.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2023-12-12T02:08:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_32': 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a8a05cc71d 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a8a05cc71d 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:42 2023 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:42 2023 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:42 2023 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:04.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0373fde7-896e-4605-8ba0-e5ef5563088e 2023-12-12T02:08:04.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_32 snap 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.export 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.export 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.export 2023-12-12T02:08:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.export 2023-12-12T02:08:04.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:04.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.export 2023-12-12T02:08:04.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.export 2023-12-12T02:08:04.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_32.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_32.export 2023-12-12T02:08:04.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_33 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_33 snap 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2023-12-12T02:08:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_33 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:08:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:06.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:06.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:06.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:06.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:06.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:06.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2023-12-12T02:08:06.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:06.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:06.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:06.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_33.status 2023-12-12T02:08:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_33.status 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.status 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_33.status 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_33.status 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:06.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2023-12-12T02:08:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:06.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2023-12-12T02:08:06.999 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5568132124749571) > weight (0.25) 2023-12-12T02:08:07.000 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3359984901792268) > weight (0.25) 2023-12-12T02:08:07.000 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2023-12-12T02:08:07.000 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2023-12-12T02:08:07.000 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-12T02:08:07.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:08:07.003+0000 7f465922d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 55247) UID: 1000 2023-12-12T02:08:07.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 183 2023-12-12T02:08:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.status 2023-12-12T02:08:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:07.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.status 2023-12-12T02:08:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2023-12-12T02:08:07.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:07.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=183 2023-12-12T02:08:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:07.324 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:07.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-12T02:08:07.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:07.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 183 = 2023-12-12T02:08:07.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:08.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-12T02:08:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 183 = 2023-12-12T02:08:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:08.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.status 2023-12-12T02:08:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2023-12-12T02:08:08.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:08.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=183 2023-12-12T02:08:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 183 = 183 2023-12-12T02:08:08.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:08.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:09.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2023-12-12T02:08:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_33': 2023-12-12T02:08:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:10.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de84b9a004 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de84b9a004 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:48 2023 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:48 2023 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:48 2023 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:10.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 35e5deae-4560-4878-ab3c-ca16cbea0129 2023-12-12T02:08:10.016 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_33 snap 2023-12-12T02:08:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:10.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.export 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.export 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.export 2023-12-12T02:08:10.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.export 2023-12-12T02:08:10.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:10.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.export 2023-12-12T02:08:10.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:10.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.export 2023-12-12T02:08:10.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_33.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_33.export 2023-12-12T02:08:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_34 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_34 snap 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:10.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:11.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:11.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:12.800 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:12.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_34 2023-12-12T02:08:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:12.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:08:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:12.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2023-12-12T02:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:12.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:12.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:12.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2023-12-12T02:08:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-12T02:08:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:12.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2023-12-12T02:08:12.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:08:12.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:12.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:08:12.967 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:12.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2023-12-12T02:08:13.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2023-12-12T02:08:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:13.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:13.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:13.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:13.079 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:08:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2023-12-12T02:08:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:13.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:13.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:13.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_34.status 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + echo cluster2-mirror-image_34.status 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:sed -e s|[/ ]|_|g 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.status 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_34.status 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_34.status 2023-12-12T02:08:13.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:13.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2023-12-12T02:08:13.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 185 2023-12-12T02:08:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.status 2023-12-12T02:08:13.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.status 2023-12-12T02:08:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2023-12-12T02:08:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:13.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=185 2023-12-12T02:08:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:13.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-12T02:08:14.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:14.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=101 2023-12-12T02:08:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 185 = 101 2023-12-12T02:08:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-12T02:08:14.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:14.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=101 2023-12-12T02:08:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 185 = 101 2023-12-12T02:08:14.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:15.001 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2023-12-12T02:08:15.001 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:08:15.001 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2023-12-12T02:08:15.001 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2023-12-12T02:08:15.002 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2023-12-12T02:08:15.004 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2023-12-12T02:08:15.004 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-12T02:08:15.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:15.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 185 = 2023-12-12T02:08:15.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.status 2023-12-12T02:08:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2023-12-12T02:08:17.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:17.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=185 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 185 = 185 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:17.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:18.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_34': 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120bbe90a4d8 2023-12-12T02:08:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120bbe90a4d8 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:00:55 2023 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:00:55 2023 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:00:55 2023 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 9d532316-c42a-45ca-a0b9-856d68476664 2023-12-12T02:08:18.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_34 snap 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.export 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.export 2023-12-12T02:08:18.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.export 2023-12-12T02:08:18.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.export 2023-12-12T02:08:18.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.export 2023-12-12T02:08:18.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.export 2023-12-12T02:08:18.906 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_34.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_34.export 2023-12-12T02:08:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_35 2023-12-12T02:08:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_35 snap 2023-12-12T02:08:18.946 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:18.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:19.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:20.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:20.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_35 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i state 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:20.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:20.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:20.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2023-12-12T02:08:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-12T02:08:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:20.361 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:20.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:08:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2023-12-12T02:08:20.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-12T02:08:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:20.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:20.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:20.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:08:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:20.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2023-12-12T02:08:20.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2023-12-12T02:08:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_35 2023-12-12T02:08:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:08:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:20.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:08:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:08:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:08:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:08:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:08:20.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:08:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:08:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:08:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:08:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_35 2023-12-12T02:08:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grep -i Replaying 2023-12-12T02:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:echo "state": "Replaying" 2023-12-12T02:08:20.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2023-12-12T02:08:20.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_35.status 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_35.status 2023-12-12T02:08:20.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.status 2023-12-12T02:08:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_35.status 2023-12-12T02:08:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_35.status 2023-12-12T02:08:20.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2023-12-12T02:08:21.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 187 2023-12-12T02:08:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.status 2023-12-12T02:08:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.status 2023-12-12T02:08:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2023-12-12T02:08:21.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:21.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=187 2023-12-12T02:08:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:21.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:22.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-12T02:08:22.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:22.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=102 2023-12-12T02:08:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 187 = 102 2023-12-12T02:08:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-12T02:08:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:22.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 187 = 2023-12-12T02:08:22.649 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:23.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-12T02:08:23.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:23.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 187 = 2023-12-12T02:08:23.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.status 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2023-12-12T02:08:25.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:25.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=187 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 187 = 187 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:25.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:25.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:26.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2023-12-12T02:08:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_35': 2023-12-12T02:08:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e92d3a3b2 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e92d3a3b2 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:08 2023 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:08 2023 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:08 2023 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: bf53f491-4b7f-44d9-bb31-ba260cb00dd6 2023-12-12T02:08:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_35 snap 2023-12-12T02:08:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:08:26.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.export 2023-12-12T02:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.export 2023-12-12T02:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.export 2023-12-12T02:08:26.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.export 2023-12-12T02:08:26.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:26.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.export 2023-12-12T02:08:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:26.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.export 2023-12-12T02:08:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_35.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_35.export 2023-12-12T02:08:26.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_36 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_36 snap 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:26.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:27.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2023-12-12T02:08:28.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_36 2023-12-12T02:08:28.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local instance=cluster1 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleepgrep 0 2023-12-12T02:08:28.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:28.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:28.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2023-12-12T02:08:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2023-12-12T02:08:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:28.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2023-12-12T02:08:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:08:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:28.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:28.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:28.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:28.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2023-12-12T02:08:29.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:29.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2023-12-12T02:08:29.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_36.status 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_36.status 2023-12-12T02:08:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.status 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_36.status 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_36.status 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:29.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:29.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2023-12-12T02:08:29.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 189 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.status 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.status 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2023-12-12T02:08:29.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:29.893 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=189 2023-12-12T02:08:29.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:29.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:30.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-12T02:08:30.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:30.167 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=103 2023-12-12T02:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 189 = 103 2023-12-12T02:08:30.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-12T02:08:30.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:30.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=103 2023-12-12T02:08:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 189 = 103 2023-12-12T02:08:30.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:31.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-12T02:08:31.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:31.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:31.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:31.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 189 = 2023-12-12T02:08:31.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:08:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:33.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.status 2023-12-12T02:08:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2023-12-12T02:08:33.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:33.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=189 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 189 = 189 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:33.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:33.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:34.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2023-12-12T02:08:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_36': 2023-12-12T02:08:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:34.472 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a8388bb6e2 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a8388bb6e2 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:14 2023 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:14 2023 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:14 2023 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 11209fbb-fa2b-470e-83df-ef0a30ead77a 2023-12-12T02:08:34.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_36 snap 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.export 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.export 2023-12-12T02:08:34.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.export 2023-12-12T02:08:34.478 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.export 2023-12-12T02:08:34.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:34.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.export 2023-12-12T02:08:34.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.export 2023-12-12T02:08:35.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_36.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_36.export 2023-12-12T02:08:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:35.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_37 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_37 snap 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:35.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:35.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:36.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_37 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:36.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:36.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:36.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:36.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:36.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2023-12-12T02:08:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:36.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:36.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:36.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_37.status 2023-12-12T02:08:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_37.status 2023-12-12T02:08:36.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.status 2023-12-12T02:08:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_37.status 2023-12-12T02:08:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_37.status 2023-12-12T02:08:36.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:36.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2023-12-12T02:08:37.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 191 2023-12-12T02:08:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.status 2023-12-12T02:08:37.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.status 2023-12-12T02:08:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2023-12-12T02:08:37.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:37.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=191 2023-12-12T02:08:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:37.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-12T02:08:37.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:37.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=104 2023-12-12T02:08:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 191 = 104 2023-12-12T02:08:37.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-12T02:08:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:38.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 191 = 2023-12-12T02:08:38.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.status 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2023-12-12T02:08:38.879 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:38.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=191 2023-12-12T02:08:38.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 191 = 191 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:38.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:39.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2023-12-12T02:08:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_37': 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10dec887ae17 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10dec887ae17 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:23 2023 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:23 2023 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:23 2023 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:40.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 2e8a2d60-c409-430d-a9f4-c3c007ec57eb 2023-12-12T02:08:40.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_37 snap 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.export 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.export 2023-12-12T02:08:40.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.export 2023-12-12T02:08:40.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.export 2023-12-12T02:08:40.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:40.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.export 2023-12-12T02:08:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:40.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.export 2023-12-12T02:08:40.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_37.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_37.export 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_38 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_38 snap 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:40.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:41.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:41.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2023-12-12T02:08:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:42.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_38 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:42.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:42.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:42.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:42.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:42.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:42.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2023-12-12T02:08:42.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:42.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:42.422 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:42.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_38.status 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_38.status 2023-12-12T02:08:42.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.status 2023-12-12T02:08:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_38.status 2023-12-12T02:08:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_38.status 2023-12-12T02:08:42.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:42.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2023-12-12T02:08:43.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 193 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.status 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.status 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2023-12-12T02:08:43.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=193 2023-12-12T02:08:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:43.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:43.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-12T02:08:43.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:43.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=105 2023-12-12T02:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 193 = 105 2023-12-12T02:08:43.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:44.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-12T02:08:44.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:44.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 193 = 2023-12-12T02:08:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.status 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2023-12-12T02:08:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:44.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=193 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 193 = 193 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:44.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:45.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2023-12-12T02:08:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_38': 2023-12-12T02:08:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:46.070 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b8e1349db 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b8e1349db 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:32 2023 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:32 2023 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:32 2023 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 3fd2a8e9-d9a7-49d9-9916-b33d7e40559b 2023-12-12T02:08:46.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:46.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_38 snap 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.export 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.export 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.export 2023-12-12T02:08:46.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.export 2023-12-12T02:08:46.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:46.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.export 2023-12-12T02:08:46.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:46.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.export 2023-12-12T02:08:46.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_38.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_38.export 2023-12-12T02:08:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_39 2023-12-12T02:08:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_39 snap 2023-12-12T02:08:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:46.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:47.830 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:48.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_39 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1grep 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:48.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:48.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:48.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:48.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2023-12-12T02:08:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2023-12-12T02:08:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:49.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2023-12-12T02:08:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:49.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:49.129 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_39.status 2023-12-12T02:08:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_39.status 2023-12-12T02:08:49.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.status 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_39.status 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_39.status 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:49.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2023-12-12T02:08:49.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 195 2023-12-12T02:08:49.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.status 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.status 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2023-12-12T02:08:49.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:49.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=195 2023-12-12T02:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:49.931 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-12T02:08:50.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:50.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=106 2023-12-12T02:08:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 195 = 106 2023-12-12T02:08:50.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-12T02:08:50.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 195 = 2023-12-12T02:08:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:51.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-12T02:08:51.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:51.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:51.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 195 = 2023-12-12T02:08:51.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.status 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2023-12-12T02:08:53.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:53.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=195 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 195 = 195 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:54.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2023-12-12T02:08:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_39': 2023-12-12T02:08:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:08:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:08:54.310 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129ef51124bf 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129ef51124bf 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:41 2023 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:41 2023 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:41 2023 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: bdb4862d-315a-44e3-8a44-de343b3940cd 2023-12-12T02:08:54.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_39 snap 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.export 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.export 2023-12-12T02:08:54.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.export 2023-12-12T02:08:54.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.export 2023-12-12T02:08:54.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:54.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.export 2023-12-12T02:08:54.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:08:54.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.export 2023-12-12T02:08:54.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_39.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_39.export 2023-12-12T02:08:54.982 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_40 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_40 snap 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:08:54.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2023-12-12T02:08:55.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:08:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:08:55.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:08:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2023-12-12T02:08:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:56.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:08:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:08:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:08:56.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_40 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shiftgrep 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:08:56.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:08:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:08:56.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:08:56.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:08:56.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:08:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:56.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:08:56.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2023-12-12T02:08:56.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-12T02:08:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:56.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:56.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:56.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:56.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:08:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:08:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:08:56.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:08:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:56.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:08:56.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2023-12-12T02:08:56.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2023-12-12T02:08:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:08:56.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:08:56.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:08:56.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:08:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:08:56.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2023-12-12T02:08:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:56.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:08:56.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_40.status 2023-12-12T02:08:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_40.status 2023-12-12T02:08:56.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.status 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_40.status 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_40.status 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:56.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2023-12-12T02:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:56.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2023-12-12T02:08:57.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 197 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.status 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.status 2023-12-12T02:08:57.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2023-12-12T02:08:57.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:57.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:08:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=197 2023-12-12T02:08:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:57.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-12T02:08:58.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=107 2023-12-12T02:08:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 197 = 107 2023-12-12T02:08:58.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:08:58.635 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-12T02:08:58.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:58.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:58.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:58.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 197 = 2023-12-12T02:08:58.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-12T02:08:59.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:08:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:08:59.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 197 = 2023-12-12T02:08:59.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:09:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:01.195 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:09:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.status 2023-12-12T02:09:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2023-12-12T02:09:01.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=197 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 197 = 197 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:01.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:01.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:02.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2023-12-12T02:09:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_40': 2023-12-12T02:09:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:02.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a858f222eb 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a858f222eb 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:50 2023 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:50 2023 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:50 2023 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0b428cbf-b8e2-4dfc-8c1f-e99c27ead9e9 2023-12-12T02:09:02.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_40 snap 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.export 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.export 2023-12-12T02:09:02.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.export 2023-12-12T02:09:02.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.export 2023-12-12T02:09:02.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:02.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.export 2023-12-12T02:09:02.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:02.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.export 2023-12-12T02:09:02.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_40.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_40.export 2023-12-12T02:09:03.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_41 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_41 snap 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:03.227 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:03.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:04.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status+ mirror/image_41 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep+ -ishift state 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:04.230 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:09:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:09:04.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:04.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:04.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:04.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:04.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2023-12-12T02:09:04.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:04.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2023-12-12T02:09:04.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_41.status 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_41.status 2023-12-12T02:09:04.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.status 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_41.status 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_41.status 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:04.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2023-12-12T02:09:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:04.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2023-12-12T02:09:05.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 199 2023-12-12T02:09:05.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.status 2023-12-12T02:09:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.status 2023-12-12T02:09:05.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2023-12-12T02:09:05.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:05.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=199 2023-12-12T02:09:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-12T02:09:05.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:05.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=108 2023-12-12T02:09:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 199 = 108 2023-12-12T02:09:05.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:06.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-12T02:09:06.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:06.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 199 = 2023-12-12T02:09:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:06.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-12T02:09:06.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:06.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 199 = 2023-12-12T02:09:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:08.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.status 2023-12-12T02:09:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2023-12-12T02:09:08.603 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:08.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=199 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 199 = 199 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:09.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2023-12-12T02:09:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_41': 2023-12-12T02:09:09.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de147d6707 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de147d6707 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:01:59 2023 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:01:59 2023 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:01:59 2023 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:09.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: aee6cea0-e960-4a62-b9b4-73a03085ac96 2023-12-12T02:09:09.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_41 snap 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.export 2023-12-12T02:09:09.741 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.export 2023-12-12T02:09:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.export 2023-12-12T02:09:09.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.export 2023-12-12T02:09:09.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:09.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.export 2023-12-12T02:09:10.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.export 2023-12-12T02:09:10.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_41.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_41.export 2023-12-12T02:09:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_42 2023-12-12T02:09:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_42 snap 2023-12-12T02:09:10.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2023-12-12T02:09:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:11.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:12.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_42 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state+ 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:local cluster=cluster1 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:09:12.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:12.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:12.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:12.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:12.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:12.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2023-12-12T02:09:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-12T02:09:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:12.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_42 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:12.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:12.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:12.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_42 2023-12-12T02:09:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_42'] 2023-12-12T02:09:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:12.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_42 2023-12-12T02:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:12.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:12.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:12.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:12.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_42 2023-12-12T02:09:12.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:12.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:12.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:12.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_42.status 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_42.status 2023-12-12T02:09:12.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.status 2023-12-12T02:09:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_42.status 2023-12-12T02:09:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_42.status 2023-12-12T02:09:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:12.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2023-12-12T02:09:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:12.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2023-12-12T02:09:13.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 201 2023-12-12T02:09:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.status 2023-12-12T02:09:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:13.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.status 2023-12-12T02:09:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2023-12-12T02:09:13.327 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:13.379 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=201 2023-12-12T02:09:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:13.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:13.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-12T02:09:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:13.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=109 2023-12-12T02:09:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 201 = 109 2023-12-12T02:09:13.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-12T02:09:14.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:14.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:14.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 201 = 2023-12-12T02:09:14.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-12T02:09:14.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:15.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:15.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 201 = 2023-12-12T02:09:15.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:16.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.status 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2023-12-12T02:09:16.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:16.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=201 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 201 = 201 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:17.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_42': 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b4037ba0b 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b4037ba0b 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:09 2023 2023-12-12T02:09:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:09 2023 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:09 2023 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 82ea7790-1532-430b-aec5-73538850cd6e 2023-12-12T02:09:17.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_42 snap 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:17.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.export 2023-12-12T02:09:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.export 2023-12-12T02:09:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.export 2023-12-12T02:09:17.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.export 2023-12-12T02:09:18.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.export 2023-12-12T02:09:18.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:18.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.export 2023-12-12T02:09:18.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_42.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_42.export 2023-12-12T02:09:18.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_43 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_43 snap 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:19.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:19.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:20.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_43 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ localgrep instance=cluster1 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state+ 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:local loop_instance 2023-12-12T02:09:20.333 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:20.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:20.336 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2023-12-12T02:09:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-12T02:09:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:20.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2023-12-12T02:09:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:20.475 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:20.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:20.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:20.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2023-12-12T02:09:20.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2023-12-12T02:09:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:20.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:20.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:20.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:20.645 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:20.646 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2023-12-12T02:09:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:20.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:20.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:20.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:20.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_43.status 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_43.status 2023-12-12T02:09:20.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.status 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_43.status 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_43.status 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:20.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:20.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2023-12-12T02:09:21.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 203 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.status 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.status 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2023-12-12T02:09:21.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=203 2023-12-12T02:09:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-12T02:09:22.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=110 2023-12-12T02:09:22.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 203 = 110 2023-12-12T02:09:22.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-12T02:09:22.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:22.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:22.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 203 = 2023-12-12T02:09:22.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:23.494 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-12T02:09:23.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:23.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:23.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 203 = 2023-12-12T02:09:23.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.status 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2023-12-12T02:09:25.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:25.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=203 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 203 = 203 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:25.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:25.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:26.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_43': 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129e23dde1a1 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129e23dde1a1 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:15 2023 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:15 2023 2023-12-12T02:09:26.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:15 2023 2023-12-12T02:09:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: a7f2974e-8b72-4697-9e38-a3cab091c892 2023-12-12T02:09:26.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_43 snap 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.export 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.export 2023-12-12T02:09:26.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.export 2023-12-12T02:09:26.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.export 2023-12-12T02:09:26.508 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.export 2023-12-12T02:09:26.859 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:26.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.export 2023-12-12T02:09:26.950 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_43.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_43.export 2023-12-12T02:09:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_44 2023-12-12T02:09:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_44 snap 2023-12-12T02:09:26.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:26.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:27.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:27.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:28.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_44 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local instance=cluster1 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ greplocal -i state loop_instance 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:28.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:28.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:28.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:28.899 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2023-12-12T02:09:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:29.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:29.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:29.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:29.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_44.status 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_44.status 2023-12-12T02:09:29.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.status 2023-12-12T02:09:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_44.status 2023-12-12T02:09:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_44.status 2023-12-12T02:09:29.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:29.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2023-12-12T02:09:29.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 205 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.status 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.status 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2023-12-12T02:09:29.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:29.957 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:29.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=205 2023-12-12T02:09:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-12T02:09:30.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:30.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=111 2023-12-12T02:09:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 205 = 111 2023-12-12T02:09:30.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-12T02:09:30.639 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:30.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=111 2023-12-12T02:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 205 = 111 2023-12-12T02:09:30.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:31.011 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6372872704992463) > weight (0.25) 2023-12-12T02:09:31.011 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.885741386068418) > weight (0.25) 2023-12-12T02:09:31.011 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-12T02:09:31.012 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-12T02:09:31.012 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-12T02:09:31.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:09:31.016+0000 7fdfe05df640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 45893) UID: 1000 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-12T02:09:31.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:31.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 205 = 2023-12-12T02:09:31.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.status 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2023-12-12T02:09:33.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:33.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=205 2023-12-12T02:09:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 205 = 205 2023-12-12T02:09:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2023-12-12T02:09:33.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:33.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:34.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_44': 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a85ced318a 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a85ced318a 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:34.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:27 2023 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:27 2023 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:27 2023 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 0740d897-205d-4bac-ae1f-efbbf3092f25 2023-12-12T02:09:34.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_44 snap 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.export 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.export 2023-12-12T02:09:34.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.export 2023-12-12T02:09:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.export 2023-12-12T02:09:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:34.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.export 2023-12-12T02:09:34.847 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:34.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.export 2023-12-12T02:09:35.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_44.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_44.export 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_45 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_45 snap 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:35.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:36.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:36.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:37.013 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-12T02:09:37.013 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:09:37.013 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-12T02:09:37.013 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-12T02:09:37.014 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-12T02:09:37.017 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-12T02:09:37.017 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons+ cluster1 rbd mirror status mirror/image_45 2023-12-12T02:09:37.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:37.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:37.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:37.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:37.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:37.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:37.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2023-12-12T02:09:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-12T02:09:37.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:37.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:37.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:37.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:37.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2023-12-12T02:09:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-12T02:09:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:37.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:37.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:37.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:37.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2023-12-12T02:09:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2023-12-12T02:09:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:37.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2023-12-12T02:09:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:09:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:09:37.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:37.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:09:37.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:37.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2023-12-12T02:09:37.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:37.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:37.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_45.status 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_45.status 2023-12-12T02:09:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.status 2023-12-12T02:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_45.status 2023-12-12T02:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_45.status 2023-12-12T02:09:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2023-12-12T02:09:38.005 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-12T02:09:38.005 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-12T02:09:38.005 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-12T02:09:38.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:09:38.012+0000 7f632031b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 41981) UID: 1000 2023-12-12T02:09:38.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 207 2023-12-12T02:09:38.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.status 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.status 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2023-12-12T02:09:38.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=207 2023-12-12T02:09:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:38.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:38.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-12T02:09:38.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:38.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=112 2023-12-12T02:09:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 207 = 112 2023-12-12T02:09:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:39.006 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-12T02:09:39.007 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:09:39.007 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-12T02:09:39.007 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T02:09:39.007 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T02:09:39.010 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T02:09:39.010 INFO:tasks.rbd_mirror_thrash:waiting for 118 secs before thrashing 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-12T02:09:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:39.084 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:39.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 207 = 2023-12-12T02:09:39.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:39.888 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-12T02:09:39.889 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:39.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:39.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 207 = 2023-12-12T02:09:39.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:09:41.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.status 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2023-12-12T02:09:41.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:41.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=207 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 207 = 207 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:41.670 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:42.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_45': 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10deeaf799d3 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10deeaf799d3 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:42.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:37 2023 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:37 2023 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:37 2023 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: b49a1ee9-32a6-4049-9eef-ab8b4ccd2069 2023-12-12T02:09:42.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_45 snap 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.export 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.export 2023-12-12T02:09:42.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.export 2023-12-12T02:09:42.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.export 2023-12-12T02:09:42.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:42.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.export 2023-12-12T02:09:43.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.export 2023-12-12T02:09:43.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_45.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_45.export 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_46 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_46 snap 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:43.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:43.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2023-12-12T02:09:44.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:44.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:45.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_46 2023-12-12T02:09:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:09:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster=cluster1 2023-12-12T02:09:45.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:09:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instancegrep 2023-12-12T02:09:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -i state 2023-12-12T02:09:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:45.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:45.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:45.268 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:45.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:45.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:45.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2023-12-12T02:09:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:09:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:09:45.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:09:45.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:45.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_46.status 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_46.status 2023-12-12T02:09:45.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.status 2023-12-12T02:09:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_46.status 2023-12-12T02:09:45.419 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_46.status 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:45.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2023-12-12T02:09:46.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 209 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.status 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.status 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2023-12-12T02:09:46.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:46.359 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:09:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=209 2023-12-12T02:09:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:46.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-12T02:09:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=113 2023-12-12T02:09:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 209 = 113 2023-12-12T02:09:46.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:09:47.035 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-12T02:09:47.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:09:47.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 209 = 2023-12-12T02:09:47.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.status 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2023-12-12T02:09:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:09:47.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:09:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=209 2023-12-12T02:09:47.988 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 209 = 209 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:47.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:48.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2023-12-12T02:09:49.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_46': 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 120b5a54619d 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.120b5a54619d 2023-12-12T02:09:49.058 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:44 2023 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:44 2023 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:44 2023 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:09:49.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:09:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:09:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 1e818293-2177-4212-8304-5c285074449e 2023-12-12T02:09:49.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_46 snap 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.export 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.export 2023-12-12T02:09:49.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.export 2023-12-12T02:09:49.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.export 2023-12-12T02:09:49.279 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:49.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.export 2023-12-12T02:09:49.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:09:49.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.export 2023-12-12T02:09:49.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_46.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_46.export 2023-12-12T02:09:49.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_47 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_47 snap 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:09:49.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:50.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:09:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:09:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:09:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:09:51.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:09:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:09:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_47 2023-12-12T02:09:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:09:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ greplocal -i cluster=cluster1 state 2023-12-12T02:09:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:09:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:09:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:09:51.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:09:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:51.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:51.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:51.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:51.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:51.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-12T02:09:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:51.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:51.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:51.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:51.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:51.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:51.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-12T02:09:51.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:51.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:51.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-12T02:09:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:51.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:51.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:51.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:09:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:51.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:51.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-12-12T02:09:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:51.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:51.820 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:09:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:52.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-12T02:09:52.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:52.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-12T02:09:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:52.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-12T02:09:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:52.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:52.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:52.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:52.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:53.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-12T02:09:53.112 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:53.113 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-12T02:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:53.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:53.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:53.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:53.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-12T02:09:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:53.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:53.249 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:53.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:53.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:09:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:53.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-12-12T02:09:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:53.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:53.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:09:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:55.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-12T02:09:55.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:55.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:55.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:55.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:55.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-12T02:09:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:55.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:55.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:55.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:09:55.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:09:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:55.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:09:55.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-12T02:09:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:55.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-12T02:09:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:09:55.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:09:55.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:09:55.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:09:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:55.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:09:55.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-12T02:09:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:55.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:55.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:55.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:09:55.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:09:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:09:55.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-12-12T02:09:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:09:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:09:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:09:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:09:59.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:59.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:09:59.959 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:09:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:09:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:09:59.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-12T02:10:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:10:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:00.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-12T02:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:00.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:10:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:00.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:10:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-12T02:10:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:10:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:00.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2023-12-12T02:10:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:10:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:00.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:10:00.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:00.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:00.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:10:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:00.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2023-12-12T02:10:00.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:10:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:00.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:00.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:00.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:10:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:00.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2023-12-12T02:10:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:10:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:00.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:00.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:10:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:10:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:10:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2023-12-12T02:10:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:10:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:10:08.523 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:08.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:10:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:08.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2023-12-12T02:10:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2023-12-12T02:10:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:08.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:08.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:08.668 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:10:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:08.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2023-12-12T02:10:08.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:10:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:10:08.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:10:08.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:08.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_47.status 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_47.status 2023-12-12T02:10:08.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.status 2023-12-12T02:10:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_47.status 2023-12-12T02:10:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_47.status 2023-12-12T02:10:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2023-12-12T02:10:08.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:08.819 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2023-12-12T02:10:09.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 211 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.status 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.status 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2023-12-12T02:10:09.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:09.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:10:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=211 2023-12-12T02:10:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:09.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:10:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:09.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:09.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-12T02:10:09.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=114 2023-12-12T02:10:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 211 = 114 2023-12-12T02:10:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:10:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:10.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-12T02:10:10.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:10.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:10.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 211 = 2023-12-12T02:10:10.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:10:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:10.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-12T02:10:10.897 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:10.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:10.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:10.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 211 = 2023-12-12T02:10:10.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.status 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2023-12-12T02:10:12.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:12.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=211 2023-12-12T02:10:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 211 = 211 2023-12-12T02:10:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2023-12-12T02:10:12.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:12.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:13.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_47': 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 129eb315bfda 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.129eb315bfda 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:10:13.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:02:53 2023 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:02:53 2023 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:02:53 2023 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: f7b0d429-ffe2-491a-861c-6f5dbcb6ae73 2023-12-12T02:10:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:10:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_47 snap 2023-12-12T02:10:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:10:13.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:10:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.export 2023-12-12T02:10:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.export 2023-12-12T02:10:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.export 2023-12-12T02:10:13.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.export 2023-12-12T02:10:13.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.export 2023-12-12T02:10:14.275 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:14.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.export 2023-12-12T02:10:14.368 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_47.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_47.export 2023-12-12T02:10:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_48 2023-12-12T02:10:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_48 snap 2023-12-12T02:10:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:14.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:14.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:10:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:10:16.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:10:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_48 2023-12-12T02:10:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + local cluster_instance=cluster1 2023-12-12T02:10:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:10:16.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:10:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:10:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:10:16.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:16.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:16.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:10:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:16.316 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2023-12-12T02:10:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-12T02:10:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:16.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:16.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:16.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:16.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:10:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2023-12-12T02:10:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-12T02:10:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:10:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:16.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:16.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2023-12-12T02:10:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2023-12-12T02:10:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:16.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:3 rbd mirror status mirror/image_48 2023-12-12T02:10:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:10:16.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 cluster instance 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:10:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:3 cluster1 2023-12-12T02:10:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:3 2023-12-12T02:10:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:3 local_cluster instance 2023-12-12T02:10:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:3 2023-12-12T02:10:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:16.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 3 = cluster1:3 ] 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=3 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=3 2023-12-12T02:10:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.3 admin socket 2023-12-12T02:10:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2023-12-12T02:10:16.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_48 2023-12-12T02:10:16.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:10:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:10:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:10:16.872 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:10:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:10:16.873 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:16.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_48.status 2023-12-12T02:10:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_48.status 2023-12-12T02:10:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:16.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.status 2023-12-12T02:10:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_48.status 2023-12-12T02:10:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_48.status 2023-12-12T02:10:16.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:16.877 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2023-12-12T02:10:17.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 213 2023-12-12T02:10:17.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.status 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.status 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2023-12-12T02:10:17.930 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:10:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=213 2023-12-12T02:10:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:17.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:10:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:18.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-12T02:10:18.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:18.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=115 2023-12-12T02:10:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 213 = 115 2023-12-12T02:10:18.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:10:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:18.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-12T02:10:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=115 2023-12-12T02:10:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 213 = 115 2023-12-12T02:10:18.733 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-12T02:10:19.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:19.604 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 213 = 2023-12-12T02:10:19.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.status 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2023-12-12T02:10:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:21.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=213 2023-12-12T02:10:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 213 = 213 2023-12-12T02:10:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:21.277 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_48': 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12a82a512438 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12a82a512438 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:10:22.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:03:01 2023 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:03:01 2023 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:03:01 2023 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: dea4476e-358a-4185-8c03-a70bae8584fb 2023-12-12T02:10:22.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:10:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_48 snap 2023-12-12T02:10:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:10:22.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:10:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.export 2023-12-12T02:10:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.export 2023-12-12T02:10:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.export 2023-12-12T02:10:22.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.export 2023-12-12T02:10:22.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:22.562 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.export 2023-12-12T02:10:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.export 2023-12-12T02:10:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_48.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_48.export 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_49 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_49 snap 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:22.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:23.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2023-12-12T02:10:24.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_49 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep -i state 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance=cluster1 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:10:24.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.273 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:24.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:10:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:24.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2023-12-12T02:10:24.407 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-12T02:10:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:24.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 cluster instance 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:1 cluster1 2023-12-12T02:10:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:1 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:1 local_cluster instance 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:1 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:24.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 1 = cluster1:1 ] 2023-12-12T02:10:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=1 2023-12-12T02:10:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=1 2023-12-12T02:10:24.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.1 admin socket 2023-12-12T02:10:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:24.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2023-12-12T02:10:24.435 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2023-12-12T02:10:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2023-12-12T02:10:24.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd mirror status : get status for rbd mirror 2023-12-12T02:10:24.555 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:admin_socket: invalid command 2023-12-12T02:10:24.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 cluster instance 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:24.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:2 cluster1 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:2 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:2 local_cluster instance 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:2 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 2 = cluster1:2 ] 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:24.558 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=2 2023-12-12T02:10:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=2 2023-12-12T02:10:24.559 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.2 admin socket 2023-12-12T02:10:24.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2023-12-12T02:10:24.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2023-12-12T02:10:24.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:10:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:10:24.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:10:24.704 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:10:24.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:24.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_49.status 2023-12-12T02:10:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_49.status 2023-12-12T02:10:24.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.status 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_49.status 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_49.status 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:24.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2023-12-12T02:10:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:24.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2023-12-12T02:10:25.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 215 2023-12-12T02:10:25.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.status 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.status 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2023-12-12T02:10:25.309 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:25.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:10:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=215 2023-12-12T02:10:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:25.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:25.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-12T02:10:25.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:25.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=116 2023-12-12T02:10:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 215 = 116 2023-12-12T02:10:25.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-12T02:10:26.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:26.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 215 = 2023-12-12T02:10:26.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-12T02:10:26.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:27.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:27.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 215 = 2023-12-12T02:10:27.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:10:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.status 2023-12-12T02:10:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2023-12-12T02:10:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=215 2023-12-12T02:10:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 215 = 215 2023-12-12T02:10:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:29.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_49': 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 10de4a0714c6 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.10de4a0714c6 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:10:29.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:03:08 2023 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:03:08 2023 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:03:08 2023 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 50176e92-488b-4d3d-8b4d-6f801bfbf83c 2023-12-12T02:10:29.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:10:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_49 snap 2023-12-12T02:10:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.export 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.export 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.export 2023-12-12T02:10:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.export 2023-12-12T02:10:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:29.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.export 2023-12-12T02:10:30.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:30.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.export 2023-12-12T02:10:30.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_49.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_49.export 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_50 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_snap cluster2 mirror image_50 snap 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:30.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2023-12-12T02:10:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2023-12-12T02:10:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:31.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=started 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:31.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=started 2023-12-12T02:10:32.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_result 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=stopped 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemons cluster1 rbd mirror status mirror/image_50 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_instance=cluster1 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + shift 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:grep+ -ilocal state instance=cluster1 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loop_instance 2023-12-12T02:10:32.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 != cluster1 ] 2023-12-12T02:10:32.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 cluster instance 2023-12-12T02:10:32.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ daemon_asok_file cluster1:0 cluster1 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1:0 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1:0 local_cluster instance 2023-12-12T02:10:32.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1:0 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=local_cluster 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 = cluster1:0 ] 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval local_cluster=cluster1 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local_cluster=cluster1 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:10:32.308 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph-conf --cluster cluster1 --name client.mirror.0 admin socket 2023-12-12T02:10:32.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2023-12-12T02:10:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2023-12-12T02:10:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:32.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ status_result= "state": "Replaying" 2023-12-12T02:10:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo "state": "Replaying" 2023-12-12T02:10:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -i Replaying 2023-12-12T02:10:32.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: "state": "Replaying" 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=started 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test started = started 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = journal ] 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ snapshot = snapshot ] 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_cluster=cluster1 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:32.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster2-mirror-image_50.status 2023-12-12T02:10:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster2-mirror-image_50.status 2023-12-12T02:10:32.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local status_log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.status 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mkfname cluster1-mirror-image_50.status 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo cluster1-mirror-image_50.status 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -e s|[/ ]|_|g 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local local_status_log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:32.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2023-12-12T02:10:33.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Snapshot ID: 217 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.status 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.status 2023-12-12T02:10:33.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2023-12-12T02:10:33.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:33.398 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/id 2023-12-12T02:10:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snapshot_id=217 2023-12-12T02:10:33.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:33.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.2 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-12T02:10:33.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:33.669 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=117 2023-12-12T02:10:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 217 = 117 2023-12-12T02:10:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.4 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-12T02:10:34.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:34.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 217 = 2023-12-12T02:10:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.8 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:34.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-12T02:10:34.948 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:10:35.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:-:1.1: Document is empty 2023-12-12T02:10:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: 2023-12-12T02:10:35.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:^ 2023-12-12T02:10:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id= 2023-12-12T02:10:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 217 = 2023-12-12T02:10:35.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1.6 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local log=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.status 2023-12-12T02:10:36.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2023-12-12T02:10:36.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -c //snapshots/snapshot[namespace/complete='true' and position()=last()] 2023-12-12T02:10:36.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2023-12-12T02:10:36.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local primary_snapshot_id=217 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 217 = 217 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:10:36.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:10:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2023-12-12T02:10:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_50': 2023-12-12T02:10:37.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 128 MiB in 32 objects 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 2 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 12be8ca5a461 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.12be8ca5a461 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering, non-primary 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:03:16 2023 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:03:16 2023 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:03:16 2023 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: protected: False 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:10:37.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 7a672f15-d393-4344-a316-8df164a5da70 2023-12-12T02:10:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: false 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ compare_image_snaps mirror image_50 snap 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap_name=snap 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local ret=0 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local rmt_export=/tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.export 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local loc_export=/tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.export 2023-12-12T02:10:37.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.export 2023-12-12T02:10:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.export 2023-12-12T02:10:38.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.export 2023-12-12T02:10:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2023-12-12T02:10:38.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cmp /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.export 2023-12-12T02:10:38.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -f /tmp/tmp.6PQoIp3L40/cluster2-mirror-image_50.export /tmp/tmp.6PQoIp3L40/cluster1-mirror-image_50.export 2023-12-12T02:10:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: delete many images 2023-12-12T02:10:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T02:10:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T02:10:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 02:10:38 TEST: delete many images 2023-12-12T02:10:38.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 02:10:38 TEST: delete many images 2023-12-12T02:10:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 50 2023-12-12T02:10:38.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_1 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:38.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2023-12-12T02:10:39.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-12T02:10:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:10:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:10:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:10:39.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:10:39.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:10:39.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:10:39.614+0000 7fc946507640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d296680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:39.609 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:10:39.614+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2c74e780 [3/b834df41-6596-47b9-95f6-b6243a57f231] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_2 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:40.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2023-12-12T02:10:40.943 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T02:10:40.944+0000 7f97b9f54640 -1 reset not still connected to 0x560d14f4b110 2023-12-12T02:10:40.945 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T02:10:40.944+0000 7f97b9f54640 -1 reset not still connected to 0x560d1b046000 2023-12-12T02:10:40.946 INFO:tasks.ceph.cluster1.osd.1.smithi055.stderr:2023-12-12T02:10:40.944+0000 7f97b9f54640 -1 reset not still connected to 0x560d1dd91ad0 2023-12-12T02:10:41.272 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:41.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:10:41.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:10:41.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:10:41.544 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:41.550+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d12a6b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:41.545 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:41.550+0000 7fde34ff9640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777400 [3/d1652be7-048c-4b97-9aef-5e3f0ab7f525] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:42.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_3 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2023-12-12T02:10:43.297 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:43.313 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-12T02:10:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:10:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:43.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:10:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:10:43.315 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:10:43.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:10:43.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:10:43.542+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128f9b5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:43.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:10:43.542+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f109680 [3/489d4631-b2bd-47cc-9fa0-9d572c181d98] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_4 2023-12-12T02:10:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2023-12-12T02:10:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:44.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:10:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:44.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2023-12-12T02:10:45.325 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:45.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-12T02:10:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:10:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:45.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:10:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:10:45.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:10:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:10:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:45.650+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d12a7040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:45.647 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:45.650+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e58a00 [3/6fcc8ecf-2681-4d44-82de-245f7523ff13] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:46.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_5 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:46.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2023-12-12T02:10:47.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:47.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:10:47.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:10:47.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:10:47.599 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:47.602+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d195fba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:47.600 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:47.602+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e58280 [3/c945b710-ea44-42b4-9616-704aab1766b4] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:48.672 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_6 2023-12-12T02:10:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2023-12-12T02:10:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:48.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:10:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:48.676 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2023-12-12T02:10:49.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-12T02:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:49.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_6 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:10:49.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-12T02:10:49.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-12T02:10:49.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:10:49.634+0000 7fc946507640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d296820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:49.631 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:10:49.634+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eb900 [3/0e8bda32-db29-49f2-b777-53035364d63f] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_7 2023-12-12T02:10:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2023-12-12T02:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2023-12-12T02:10:50.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-12T02:10:50.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:51.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:10:51.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:10:51.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:10:51.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:51.222+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d12a6680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:51.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:51.222+0000 7fde34ff9640 -1 rbd::mirror::ImageReplayer: 0x55a1d0776c80 [3/ce49be9c-8844-4083-8907-bd68b7a39229] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:52.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_8 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:10:52.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:52.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2023-12-12T02:10:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:52.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:10:52.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:10:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:10:53.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:10:53.221+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128f9b5ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:53.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:10:53.221+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f109400 [3/342b1cab-88ee-458a-94d9-2ad59864a055] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_9 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:54.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2023-12-12T02:10:54.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-12T02:10:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:10:54.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:10:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:54.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:10:54.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:10:55.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:10:55.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:10:55.253+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574619d2b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:55.249 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:10:55.253+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461169680 [3/55c1b0cd-97fd-4de6-bd65-a62b4f88d578] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:56.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_10 2023-12-12T02:10:56.234 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2023-12-12T02:10:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:10:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:56.235 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2023-12-12T02:10:56.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-12T02:10:56.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:10:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:10:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:10:56.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:10:57.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:10:57.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:10:57.249+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55746181a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:57.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:10:57.249+0000 7fc9b799c640 -1 rbd::mirror::ImageReplayer: 0x557461168f00 [3/73a5838f-6323-4afc-bd11-554e7ecf7c56] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:10:58.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_11 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:10:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:10:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2023-12-12T02:10:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:10:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_11 2023-12-12T02:10:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:10:58.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:10:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_11 2023-12-12T02:10:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:10:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:10:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:10:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2023-12-12T02:10:59.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2023-12-12T02:10:59.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:59.249+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d195fd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:10:59.246 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:10:59.249+0000 7fde34ff9640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777180 [3/dfbf3fd7-eb15-4e16-a0ac-43c2a8e3e631] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:00.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_12 2023-12-12T02:11:00.287 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2023-12-12T02:11:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:11:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2023-12-12T02:11:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_12 2023-12-12T02:11:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:11:00.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_12 2023-12-12T02:11:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:11:00.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2023-12-12T02:11:01.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2023-12-12T02:11:01.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:01.257+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d2969c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:01.255 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:01.257+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eb180 [3/ddfde94a-a349-4e2a-ae4d-72258f5591a6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:02.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_13 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:02.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2023-12-12T02:11:02.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_13 2023-12-12T02:11:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:11:03.005 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_13 2023-12-12T02:11:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:11:03.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2023-12-12T02:11:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2023-12-12T02:11:03.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:03.297+0000 7fc946507640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d296b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:03.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:03.297+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5ea780 [3/6c841662-f2e7-4c7e-8c09-62dd32e1eff8] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_14 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2023-12-12T02:11:05.008 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_14 2023-12-12T02:11:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:05.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:11:05.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_14 2023-12-12T02:11:05.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:11:05.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2023-12-12T02:11:05.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2023-12-12T02:11:05.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:05.353+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d0dc8000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:05.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:05.353+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777b80 [3/c9956d0a-2c28-4015-9421-074ee875e996] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:06.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_15 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:06.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2023-12-12T02:11:07.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_15 2023-12-12T02:11:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:07.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_15 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:11:07.053 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2023-12-12T02:11:07.141 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2023-12-12T02:11:07.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:07.321+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa3ed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:07.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:07.321+0000 7f596373e640 -1 rbd::mirror::ImageReplayer: 0x56128f109900 [3/09d8c9b1-86e1-481e-8a12-992f2d80f06d] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:08.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_16 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:08.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2023-12-12T02:11:08.846 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e1de4680 2023-12-12T02:11:08.846 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e4510000 2023-12-12T02:11:08.847 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e4834750 2023-12-12T02:11:08.847 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e4835040 2023-12-12T02:11:08.847 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e48356c0 2023-12-12T02:11:08.847 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:11:08.848+0000 7f21394c6640 -1 reset not still connected to 0x5606e5c09380 2023-12-12T02:11:09.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_16 2023-12-12T02:11:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:11:09.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_16 2023-12-12T02:11:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:09.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:11:09.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2023-12-12T02:11:09.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2023-12-12T02:11:09.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:09.381+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d296d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:09.377 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:09.381+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eac80 [3/cf75936b-449b-47ab-8ea6-3647877a3e66] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:10.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:10.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_17 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:10.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2023-12-12T02:11:10.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:10.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_17 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_17 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:11:10.938 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2023-12-12T02:11:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2023-12-12T02:11:11.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:11.345+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d296ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:11.342 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:11.345+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eaf00 [3/d06e3eb6-22a3-4ffe-88ca-04ba98f63457] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:12.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_18 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2023-12-12T02:11:13.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_18 2023-12-12T02:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:11:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_18 2023-12-12T02:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:11:13.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2023-12-12T02:11:13.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2023-12-12T02:11:13.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:13.361+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55746179dd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:13.358 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:13.361+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461168780 [3/2db15675-ec89-40f1-b944-460662d253f6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:14.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_19 2023-12-12T02:11:14.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2023-12-12T02:11:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:11:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:14.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2023-12-12T02:11:15.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_19 2023-12-12T02:11:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:11:15.116 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_19 2023-12-12T02:11:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:11:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2023-12-12T02:11:15.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2023-12-12T02:11:15.421 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:15.425+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55746181a1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:15.422 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:15.425+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461168a00 [3/963f10a2-9317-4e5e-b35a-fc335be2a14a] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:16.274 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_20 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:16.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2023-12-12T02:11:17.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_20 2023-12-12T02:11:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:11:17.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_20 2023-12-12T02:11:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:11:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2023-12-12T02:11:17.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2023-12-12T02:11:17.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:17.421+0000 7fc946507640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d297040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:17.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:17.421+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eaa00 [3/74a54c46-73ec-4d02-aaa0-86f3e226c449] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_21 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:18.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2023-12-12T02:11:19.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_21 2023-12-12T02:11:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:11:19.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_21 2023-12-12T02:11:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:11:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2023-12-12T02:11:19.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2023-12-12T02:11:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:19.409+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2ce52ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:19.408 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:19.409+0000 7fc94cd14640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5ea500 [3/59ede7a8-6346-4103-a0a4-f50800c189ac] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:20.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_22 2023-12-12T02:11:20.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2023-12-12T02:11:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:11:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:20.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2023-12-12T02:11:20.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_22 2023-12-12T02:11:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:20.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:11:20.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_22 2023-12-12T02:11:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:11:20.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2023-12-12T02:11:21.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2023-12-12T02:11:21.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:21.241+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2d2971e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:21.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:21.241+0000 7fc94d515640 -1 rbd::mirror::ImageReplayer: 0x55ba2c74e280 [3/e51faaaf-277b-490b-9e12-ad199fdee9c2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:22.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_23 2023-12-12T02:11:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2023-12-12T02:11:22.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2023-12-12T02:11:23.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_23 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:23.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_23 2023-12-12T02:11:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:11:23.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2023-12-12T02:11:23.300 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2023-12-12T02:11:23.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:23.657+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d12afba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:23.655 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:23.657+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0776500 [3/a4a6e595-cc0c-4cac-b030-d274e817dada] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:24.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:24.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_24 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:24.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2023-12-12T02:11:25.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:25.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_24 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_24 2023-12-12T02:11:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:11:25.172 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2023-12-12T02:11:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2023-12-12T02:11:25.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:25.521+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55746181a000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:25.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:25.521+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461954500 [3/d213a902-dc35-4507-886a-3f4b1129d6d2] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:26.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_25 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:11:26.260 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:26.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2023-12-12T02:11:27.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_25 2023-12-12T02:11:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:11:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_25 2023-12-12T02:11:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:27.197 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:11:27.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2023-12-12T02:11:27.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2023-12-12T02:11:27.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:27.517+0000 7fc946d08640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ba2cda3040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:27.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:27.517+0000 7fc94cd14640 -1 rbd::mirror::ImageReplayer: 0x55ba2d5eb680 [3/a921f597-6426-4126-a233-e3c16f3533b3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:28.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_26 2023-12-12T02:11:28.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2023-12-12T02:11:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:11:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:28.477 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2023-12-12T02:11:29.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_26 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_26 2023-12-12T02:11:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:11:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2023-12-12T02:11:29.305 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2023-12-12T02:11:29.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:29.452+0000 7fc9b1990640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574619d3860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:29.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:29.452+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461169900 [3/26f2fa83-50d9-4f27-b56d-3181fdc135a5] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:30.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_27 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:30.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2023-12-12T02:11:30.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:30.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_27 2023-12-12T02:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:11:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_27 2023-12-12T02:11:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:11:30.997 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2023-12-12T02:11:31.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2023-12-12T02:11:31.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:31.240+0000 7fc9b1990640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557461aa7860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:31.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:31.240+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461169180 [3/82b98cb4-5a2a-4924-8f4e-9681fc9a7a67] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:32.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_28 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2023-12-12T02:11:32.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:32.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_28 2023-12-12T02:11:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:11:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:32.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_28 2023-12-12T02:11:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:11:32.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2023-12-12T02:11:33.050 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2023-12-12T02:11:33.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:33.204+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128f4cb1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:33.203 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:33.204+0000 7f596373e640 -1 rbd::mirror::ImageReplayer: 0x56128f878280 [3/c9a7b9aa-ebf2-4ce9-8f2a-36162c41d7cf] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:34.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_29 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:34.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2023-12-12T02:11:34.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_29 2023-12-12T02:11:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:11:34.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_29 2023-12-12T02:11:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:11:34.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2023-12-12T02:11:35.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.513548745182247) > weight (0.25) 2023-12-12T02:11:35.018 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T02:11:35.018 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T02:11:35.018 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2023-12-12T02:11:35.019 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:11:35.020+0000 7fda7e11b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 54006) UID: 1000 2023-12-12T02:11:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2023-12-12T02:11:35.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:35.280+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d12aed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:35.277 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:35.280+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0776280 [3/d3da7272-827d-4801-89f4-6e6b377cdca7] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:36.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_30 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2023-12-12T02:11:36.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_30 2023-12-12T02:11:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:11:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_30 2023-12-12T02:11:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:11:37.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2023-12-12T02:11:37.011 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9568096794412971) > weight (0.25) 2023-12-12T02:11:37.011 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T02:11:37.011 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T02:11:37.012 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2023-12-12T02:11:37.012 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:11:37.016+0000 7fc94dd16640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 53457) UID: 1000 2023-12-12T02:11:37.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2023-12-12T02:11:37.307 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:37.308+0000 7fde347f8640 -1 rbd::mirror::ImageMap: 0x55a1d0c06fc0 handle_peer_ack_remove: failed to remove global_image_id=37a61d89-b375-4db8-b09d-d9947b095e5c 2023-12-12T02:11:38.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_31 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:38.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2023-12-12T02:11:39.001 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:39.019 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T02:11:39.019 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:11:39.019 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T02:11:39.020 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T02:11:39.020 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T02:11:39.022 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T02:11:39.022 INFO:tasks.rbd_mirror_thrash:waiting for 75 secs before thrashing 2023-12-12T02:11:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_31 2023-12-12T02:11:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:11:39.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_31 2023-12-12T02:11:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:11:39.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2023-12-12T02:11:39.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2023-12-12T02:11:39.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:39.324+0000 7fde34ff9640 -1 rbd::mirror::ImageMap: 0x55a1d0c06fc0 handle_peer_ack_remove: failed to remove global_image_id=f5bbdd5f-8443-4677-b3b4-d498845c7bef 2023-12-12T02:11:40.286 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_32 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:40.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2023-12-12T02:11:40.949 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:40.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_32 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_32 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:40.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:11:40.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2023-12-12T02:11:41.075 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2023-12-12T02:11:41.160 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:11:41.164+0000 7f2dfab1e640 -1 rbd::mirror::ImageMap: 0x55d9f74999e0 handle_peer_ack_remove: failed to remove global_image_id=0373fde7-896e-4605-8ba0-e5ef5563088e 2023-12-12T02:11:42.012 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T02:11:42.012 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:11:42.013 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T02:11:42.013 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T02:11:42.013 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T02:11:42.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T02:11:42.016 INFO:tasks.rbd_mirror_thrash:waiting for 81 secs before thrashing 2023-12-12T02:11:42.364 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_33 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:11:42.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:42.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2023-12-12T02:11:43.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_33 2023-12-12T02:11:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:11:43.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_33 2023-12-12T02:11:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:11:43.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2023-12-12T02:11:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2023-12-12T02:11:43.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:43.324+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128f9904e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:43.323 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:43.324+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f108780 [3/35e5deae-4560-4878-ab3c-ca16cbea0129] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:44.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_34 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:44.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2023-12-12T02:11:45.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_34 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:45.072 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_34 2023-12-12T02:11:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:11:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2023-12-12T02:11:45.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2023-12-12T02:11:45.360 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:45.364+0000 7fc9b118f640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557461aa7a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:45.361 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:45.364+0000 7fc9b819d640 -1 rbd::mirror::ImageReplayer: 0x557461954000 [3/9d532316-c42a-45ca-a0b9-856d68476664] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:46.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:46.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_35 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:46.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2023-12-12T02:11:47.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_35 2023-12-12T02:11:47.064 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_35 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:11:47.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2023-12-12T02:11:47.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2023-12-12T02:11:47.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:47.368+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d0f25380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:47.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:11:47.368+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e58000 [3/bf53f491-4b7f-44d9-bb31-ba260cb00dd6] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:48.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_36 2023-12-12T02:11:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2023-12-12T02:11:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:11:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:48.394 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2023-12-12T02:11:49.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_36 2023-12-12T02:11:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:49.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_36 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:11:49.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2023-12-12T02:11:49.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2023-12-12T02:11:49.254 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:11:49.256+0000 7f2dfab1e640 -1 rbd::mirror::ImageMap: 0x55d9f74999e0 handle_peer_ack_remove: failed to remove global_image_id=11209fbb-fa2b-470e-83df-ef0a30ead77a 2023-12-12T02:11:50.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_37 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:50.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2023-12-12T02:11:50.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_37 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_37 2023-12-12T02:11:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:11:50.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2023-12-12T02:11:51.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2023-12-12T02:11:51.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:51.236+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fae3d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:51.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:51.236+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f108500 [3/2e8a2d60-c409-430d-a9f4-c3c007ec57eb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:52.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_38 2023-12-12T02:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2023-12-12T02:11:52.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2023-12-12T02:11:53.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_38 2023-12-12T02:11:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:11:53.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_38 2023-12-12T02:11:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:11:53.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2023-12-12T02:11:53.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2023-12-12T02:11:53.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:53.416+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128f9b5d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:53.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:11:53.416+0000 7f596373e640 -1 rbd::mirror::ImageReplayer: 0x56128f108c80 [3/3fd2a8e9-d9a7-49d9-9916-b33d7e40559b] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:54.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_39 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:54.502 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2023-12-12T02:11:55.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_39 2023-12-12T02:11:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:11:55.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_39 2023-12-12T02:11:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:11:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2023-12-12T02:11:55.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2023-12-12T02:11:55.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:11:55.360+0000 7f2dfa31d640 -1 rbd::mirror::ImageMap: 0x55d9f74999e0 handle_peer_ack_remove: failed to remove global_image_id=bdb4862d-315a-44e3-8a44-de343b3940cd 2023-12-12T02:11:56.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_40 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:56.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2023-12-12T02:11:57.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_40 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:57.138 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_40 2023-12-12T02:11:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:11:57.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2023-12-12T02:11:57.240 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2023-12-12T02:11:57.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:57.624+0000 7fc9b1990640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574619d36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:57.622 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:57.624+0000 7fc9b799c640 -1 rbd::mirror::ImageReplayer: 0x557461168500 [3/0b428cbf-b8e2-4dfc-8c1f-e99c27ead9e9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:11:58.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:11:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:11:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_41 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:11:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2023-12-12T02:11:59.179 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:11:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_41 2023-12-12T02:11:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:11:59.202 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:11:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_41 2023-12-12T02:11:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:11:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:11:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:11:59.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2023-12-12T02:11:59.311 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2023-12-12T02:11:59.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:59.476+0000 7fc9b1990640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5574619d31e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:11:59.474 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:11:59.476+0000 7fc9b799c640 -1 rbd::mirror::ImageReplayer: 0x557461954280 [3/aee6cea0-e960-4a62-b9b4-73a03085ac96] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:00.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_42 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:00.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2023-12-12T02:12:00.970 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_42 2023-12-12T02:12:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:00.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_42 2023-12-12T02:12:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:00.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2023-12-12T02:12:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2023-12-12T02:12:01.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:01.264+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa3e9c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:01.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:01.264+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f878000 [3/82ea7790-1532-430b-aec5-73538850cd6e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:02.490 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_43 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:02.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2023-12-12T02:12:03.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_43 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_43 2023-12-12T02:12:03.190 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:03.191 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2023-12-12T02:12:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2023-12-12T02:12:03.500 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:12:03.504+0000 7f49a0dbd640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56508f778ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:03.501 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:12:03.504+0000 7f49a75ca640 -1 rbd::mirror::ImageReplayer: 0x56508fc90500 [3/a7f2974e-8b72-4697-9e38-a3cab091c892] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:04.482 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_44 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:04.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2023-12-12T02:12:05.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:05.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_44 2023-12-12T02:12:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:05.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_44 2023-12-12T02:12:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:05.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2023-12-12T02:12:05.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2023-12-12T02:12:05.502 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:05.504+0000 7f595c730640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa3f380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:05.503 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:05.504+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f108a00 [3/0740d897-205d-4bac-ae1f-efbbf3092f25] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:06.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_45 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:06.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2023-12-12T02:12:07.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_45 2023-12-12T02:12:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:07.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_45 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:07.270 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2023-12-12T02:12:07.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2023-12-12T02:12:07.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:07.528+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d11a6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:07.527 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:07.528+0000 7fde34ff9640 -1 rbd::mirror::ImageReplayer: 0x55a1d0776780 [3/b49a1ee9-32a6-4049-9eef-ab8b4ccd2069] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_46 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:08.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:08.534 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2023-12-12T02:12:09.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:09.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_46 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_46 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:09.256 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:09.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2023-12-12T02:12:09.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2023-12-12T02:12:09.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:09.779+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa83520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:09.777 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:09.779+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f108280 [3/1e818293-2177-4212-8304-5c285074449e] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:10.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_47 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:10.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2023-12-12T02:12:10.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_47 2023-12-12T02:12:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:10.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_47 2023-12-12T02:12:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:10.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2023-12-12T02:12:11.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2023-12-12T02:12:11.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:11.275+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d11a6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:11.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:11.275+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777900 [3/f7b0d429-ffe2-491a-861c-6f5dbcb6ae73] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:12.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_48 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:12.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2023-12-12T02:12:12.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_48 2023-12-12T02:12:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:12.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_48 2023-12-12T02:12:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:12.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2023-12-12T02:12:13.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2023-12-12T02:12:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:13.275+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d11a6000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:13.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:12:13.275+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e59180 [3/dea4476e-358a-4185-8c03-a70bae8584fb] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_49 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:14.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2023-12-12T02:12:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_49 2023-12-12T02:12:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_49 2023-12-12T02:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2023-12-12T02:12:15.118 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2023-12-12T02:12:15.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:15.287+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa83040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:15.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:15.287+0000 7f596373e640 -1 rbd::mirror::ImageReplayer: 0x56128f108f00 [3/50176e92-488b-4d3d-8b4d-6f801bfbf83c] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:16.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_50 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local snap=snap 2023-12-12T02:12:16.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2023-12-12T02:12:17.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing snap: 100% complete...done. 2023-12-12T02:12:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_50 2023-12-12T02:12:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:17.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_50 2023-12-12T02:12:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:17.023 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2023-12-12T02:12:17.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2023-12-12T02:12:17.274 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:17.275+0000 7f595cf31640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x56128fa82d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:12:17.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:12:17.275+0000 7f5962f3d640 -1 rbd::mirror::ImageReplayer: 0x56128f109180 [3/7a672f15-d393-4344-a316-8df164a5da70] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:12:18.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2023-12-12T02:12:18.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: image deletions should propagate 2023-12-12T02:12:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T02:12:18.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:date +%F %T 2023-12-12T02:12:18.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 02:12:18 TEST: image deletions should propagate 2023-12-12T02:12:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 02:12:18 TEST: image deletions should propagate 2023-12-12T02:12:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_images cluster1 mirror 0 2023-12-12T02:12:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.342 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_count=0 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local count 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local last_count=0 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ true 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-12T02:12:18.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep images: 2023-12-12T02:12:18.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cut -d -f 2 2023-12-12T02:12:18.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count=0 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 40 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 1 -ne 1 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster1 -p mirror mirror pool status 2023-12-12T02:12:18.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ + grepcut image health: -d 2023-12-12T02:12:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: -f 3 2023-12-12T02:12:18.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: failed to query services: (13) Permission denied 2023-12-12T02:12:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ state=OK 2023-12-12T02:12:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = ERROR 2023-12-12T02:12:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test OK = OK 2023-12-12T02:12:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:18.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 50 2023-12-12T02:12:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_1 2023-12-12T02:12:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:18.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-12T02:12:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:18.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-12T02:12:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:18.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_2 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:12:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:18.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-12T02:12:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-12T02:12:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:12:18.761 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:12:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:18.824 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_3 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.828 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:12:18.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:18.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-12T02:12:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:18.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-12T02:12:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:12:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.010 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-12T02:12:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:19.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_4 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:12:19.015 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.048 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:12:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:19.051 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-12T02:12:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:12:19.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:12:19.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_5 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:12:19.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:12:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:12:19.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.442 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-12T02:12:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:19.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:12:19.550 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-12T02:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:19.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_6 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:12:19.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.647 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:12:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:19.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.753 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:12:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_7 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:12:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:19.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-12T02:12:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:19.849 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:12:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:12:19.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.007 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-12T02:12:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:20.011 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_8 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:12:20.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:12:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-12T02:12:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:20.059 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-12T02:12:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:12:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:12:20.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_9 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:12:20.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-12T02:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:20.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.357 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:12:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:12:20.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-12T02:12:20.408 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_10 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-12T02:12:20.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:12:20.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-12T02:12:20.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:20.452 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_10 deleted 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:20.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:12:20.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_11 2023-12-12T02:12:20.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_11 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-12T02:12:20.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.641 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-12T02:12:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:20.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_11 deleted 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:20.747 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_11 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_11 2023-12-12T02:12:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_11 2023-12-12T02:12:20.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_11: (2) No such file or directory 2023-12-12T02:12:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_12 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_12 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:12:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-12T02:12:20.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.836 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-12T02:12:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:20.839 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_12 deleted 2023-12-12T02:12:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_12 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_12 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_12 2023-12-12T02:12:20.942 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:20.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_12: (2) No such file or directory 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_13 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:21.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_13 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-12T02:12:21.004 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.054 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-12T02:12:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:21.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_13 deleted 2023-12-12T02:12:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:12:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_13 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_13 2023-12-12T02:12:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_13 2023-12-12T02:12:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.185 INFO:tasks.ceph.cluster1.osd.0.smithi055.stderr:2023-12-12T02:12:21.182+0000 7f21394c6640 -1 reset not still connected to 0x5606e48356c0 2023-12-12T02:12:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_13: (2) No such file or directory 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_14 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:21.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_14 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-12T02:12:21.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-12T02:12:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:21.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_14 deleted 2023-12-12T02:12:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_14 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_14 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_14 2023-12-12T02:12:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_14: (2) No such file or directory 2023-12-12T02:12:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_15 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_15 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:12:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-12T02:12:21.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.443 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-12T02:12:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:21.447 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:21.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_15 deleted 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_15 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_15 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_15 2023-12-12T02:12:21.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_15: (2) No such file or directory 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_16 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_16 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-12T02:12:21.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.636 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-12T02:12:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:21.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_16 deleted 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_16 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.742 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_16 2023-12-12T02:12:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_16 2023-12-12T02:12:21.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.792 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_16: (2) No such file or directory 2023-12-12T02:12:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_17 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:21.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_17 2023-12-12T02:12:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:12:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-12T02:12:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:21.835 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-12T02:12:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:21.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_17 deleted 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_17 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_17 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_17 2023-12-12T02:12:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_17: (2) No such file or directory 2023-12-12T02:12:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_18 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_18 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:12:22.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-12T02:12:22.014 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.057 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-12T02:12:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:22.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_18 deleted 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:22.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_18 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_18 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_18 2023-12-12T02:12:22.163 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_18: (2) No such file or directory 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_19 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:12:22.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_19 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-12T02:12:22.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-12T02:12:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:22.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_19 deleted 2023-12-12T02:12:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:12:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_19 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_19 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_19 2023-12-12T02:12:22.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.400 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_19: (2) No such file or directory 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_20 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2023-12-12T02:12:22.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_20 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-12T02:12:22.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.441 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-12T02:12:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:22.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_20 deleted 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_20 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_20 2023-12-12T02:12:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_20 2023-12-12T02:12:22.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_20: (2) No such file or directory 2023-12-12T02:12:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_21 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_21 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:12:22.599 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-12T02:12:22.600 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-12T02:12:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:22.642 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_21 deleted 2023-12-12T02:12:22.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_21 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_21 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_21 2023-12-12T02:12:22.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_21: (2) No such file or directory 2023-12-12T02:12:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.793 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_22 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_22 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-12T02:12:22.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-12T02:12:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_22 deleted 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_22 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_22 2023-12-12T02:12:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_22 2023-12-12T02:12:22.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_22: (2) No such file or directory 2023-12-12T02:12:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_23 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_23 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:12:22.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-12T02:12:22.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.029 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-12T02:12:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:23.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_23 deleted 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_23 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_23 2023-12-12T02:12:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_23 2023-12-12T02:12:23.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_23: (2) No such file or directory 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_24 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_24 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-12T02:12:23.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-12T02:12:23.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:23.226 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_24 deleted 2023-12-12T02:12:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_24 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_24 2023-12-12T02:12:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_24 2023-12-12T02:12:23.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_24: (2) No such file or directory 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_25 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_25 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-12T02:12:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-12T02:12:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:23.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_25 deleted 2023-12-12T02:12:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:12:23.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_25 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_25 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_25 2023-12-12T02:12:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.566 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_25: (2) No such file or directory 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_26 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_26 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-12T02:12:23.572 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-12T02:12:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:23.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_26 deleted 2023-12-12T02:12:23.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_26 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_26 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_26 2023-12-12T02:12:23.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.760 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_26: (2) No such file or directory 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_27 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:23.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_27 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-12T02:12:23.767 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.804 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-12T02:12:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:23.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_27 deleted 2023-12-12T02:12:23.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_27 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_27 2023-12-12T02:12:23.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_27 2023-12-12T02:12:23.911 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_27: (2) No such file or directory 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_28 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:23.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_28 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-12T02:12:23.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-12T02:12:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:24.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_28 deleted 2023-12-12T02:12:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.148 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_28 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_28 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_28 2023-12-12T02:12:24.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_28: (2) No such file or directory 2023-12-12T02:12:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_29 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:24.199 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_29 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-12T02:12:24.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-12T02:12:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:24.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:24.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_29 deleted 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:24.340 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_29 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_29 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_29 2023-12-12T02:12:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_29: (2) No such file or directory 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_30 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_30 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-12T02:12:24.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-12T02:12:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:24.429 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_30 deleted 2023-12-12T02:12:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_30 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_30 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_30 2023-12-12T02:12:24.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_30: (2) No such file or directory 2023-12-12T02:12:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:24.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_31 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_31 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-12T02:12:24.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-12T02:12:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:24.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_31 deleted 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:24.728 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_31 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_31 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_31 2023-12-12T02:12:24.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_31: (2) No such file or directory 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_32 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_32 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-12T02:12:24.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-12T02:12:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:24.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_32 deleted 2023-12-12T02:12:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_32 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_32 2023-12-12T02:12:24.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_32 2023-12-12T02:12:24.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:24.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_32: (2) No such file or directory 2023-12-12T02:12:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:24.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_33 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_33 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-12T02:12:24.978 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-12T02:12:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:25.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_33 deleted 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:25.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_33 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_33 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_33 2023-12-12T02:12:25.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.183 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_33: (2) No such file or directory 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_34 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_34 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-12T02:12:25.189 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-12T02:12:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:25.228 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_34 deleted 2023-12-12T02:12:25.329 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_34 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_34 2023-12-12T02:12:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_34 2023-12-12T02:12:25.331 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_34: (2) No such file or directory 2023-12-12T02:12:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:25.384 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_35 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_35 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-12T02:12:25.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-12T02:12:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:25.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_35 deleted 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_35 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_35 2023-12-12T02:12:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_35 2023-12-12T02:12:25.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_35: (2) No such file or directory 2023-12-12T02:12:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:25.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_36 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_36 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.580 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:12:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-12T02:12:25.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-12T02:12:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:25.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_36 deleted 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_36 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_36 2023-12-12T02:12:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_36 2023-12-12T02:12:25.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_36: (2) No such file or directory 2023-12-12T02:12:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_37 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_37 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-12T02:12:25.776 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-12T02:12:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_37 deleted 2023-12-12T02:12:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:12:25.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_37 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_37 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_37 2023-12-12T02:12:25.921 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:25.974 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_37: (2) No such file or directory 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_38 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2023-12-12T02:12:25.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_38 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-12T02:12:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-12T02:12:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:26.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:26.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_38 deleted 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_38 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_38 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_38 2023-12-12T02:12:26.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_38: (2) No such file or directory 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_39 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.174 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_39 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-12T02:12:26.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-12T02:12:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:26.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_39 deleted 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_39 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_39 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_39 2023-12-12T02:12:26.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_39: (2) No such file or directory 2023-12-12T02:12:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:26.371 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_40 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_40 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-12T02:12:26.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-12T02:12:26.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:26.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_40 deleted 2023-12-12T02:12:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_40 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_40 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_40 2023-12-12T02:12:26.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_40: (2) No such file or directory 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_41 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_41 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-12T02:12:26.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-12T02:12:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:26.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_41 deleted 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:26.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_41 2023-12-12T02:12:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_41 2023-12-12T02:12:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_41 2023-12-12T02:12:26.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_41: (2) No such file or directory 2023-12-12T02:12:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_42 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_42 2023-12-12T02:12:26.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-12T02:12:26.765 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-12T02:12:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:26.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_42 deleted 2023-12-12T02:12:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_42 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_42 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_42 2023-12-12T02:12:26.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_42: (2) No such file or directory 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_43 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_43 2023-12-12T02:12:26.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-12T02:12:26.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:26.999 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-12T02:12:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:27.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_43 deleted 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_43 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_43 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_43 2023-12-12T02:12:27.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.152 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_43: (2) No such file or directory 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_44 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2023-12-12T02:12:27.157 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_44 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-12T02:12:27.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-12T02:12:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_44 deleted 2023-12-12T02:12:27.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_44 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.302 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_44 2023-12-12T02:12:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_44 2023-12-12T02:12:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_44: (2) No such file or directory 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_45 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2023-12-12T02:12:27.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_45 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-12T02:12:27.353 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.392 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-12T02:12:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:27.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_45 deleted 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_45 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_45 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_45 2023-12-12T02:12:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_45: (2) No such file or directory 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_46 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2023-12-12T02:12:27.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_46 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-12T02:12:27.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-12T02:12:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_46 deleted 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_46 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_46 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_46 2023-12-12T02:12:27.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.743 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_46: (2) No such file or directory 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_47 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.748 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_47 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-12T02:12:27.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-12T02:12:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_47 deleted 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_47 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_47 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_47 2023-12-12T02:12:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.947 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_47: (2) No such file or directory 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_48 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:27.952 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_48 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-12T02:12:27.953 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:27.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-12T02:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:27.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_48 deleted 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_48 2023-12-12T02:12:28.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_48 2023-12-12T02:12:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_48 2023-12-12T02:12:28.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:28.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_48: (2) No such file or directory 2023-12-12T02:12:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:28.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_49 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_49 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-12T02:12:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:28.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-12T02:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:28.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_49 deleted 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_49 2023-12-12T02:12:28.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_49 2023-12-12T02:12:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_49 2023-12-12T02:12:28.295 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:28.339 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_49: (2) No such file or directory 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_50 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:28.344 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_50 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-12T02:12:28.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:28.383 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-12T02:12:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:28.386 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_50 deleted 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_50 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_50 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_50 2023-12-12T02:12:28.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:28.535 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_50: (2) No such file or directory 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: delete images during bootstrap 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T02:12:28.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 02:12:28 TEST: delete images during bootstrap 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 02:12:28 TEST: delete images during bootstrap 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_pool_mirror_mode cluster1 mirror image 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=image 2023-12-12T02:12:28.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2023-12-12T02:12:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd: mirroring is already configured for image mode 2023-12-12T02:12:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_pool_mirror_mode cluster2 mirror image 2023-12-12T02:12:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=image 2023-12-12T02:12:28.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2023-12-12T02:12:28.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd: mirroring is already configured for image mode 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ start_mirror cluster1 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set_cluster_instance cluster1 cluster instance 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local val=cluster1 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster_var_name=cluster 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local instance_var_name=instance 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=cluster1 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ cluster1 = cluster1 ] 2023-12-12T02:12:28.622 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval cluster=cluster1 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cluster=cluster1 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ eval instance=0 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ instance=0 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=test 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 1 10 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_1 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_1 512M 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:12:28.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_1 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_1 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:12:28.675 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2023-12-12T02:12:29.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:12:29.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_1': 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 18903f9a4f4e 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.18903f9a4f4e 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:12:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:12:28 2023 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:12:28 2023 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:12:28 2023 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: b02d1b09-87f3-4241-a66b-d3349db1bca6 2023-12-12T02:12:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:12:29.380 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_1 2023-12-12T02:12:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:12:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T02:12:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:12:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:12:29.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 63982592 bytes 2023-12-12T02:12:29.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 65301504 bytes 2023-12-12T02:12:29.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 92677120 bytes 2023-12-12T02:12:30.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 172709888 bytes 2023-12-12T02:12:30.358 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:12:30.355+0000 7fd7ab621640 -1 reset not still connected to 0x557473adb790 2023-12-12T02:12:30.358 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:12:30.355+0000 7fd7ab621640 -1 reset not still connected to 0x557483f7e410 2023-12-12T02:12:30.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 191753216 bytes 2023-12-12T02:12:30.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 235034624 bytes 2023-12-12T02:12:30.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 255435776 bytes 2023-12-12T02:12:30.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 313766912 bytes 2023-12-12T02:12:31.259 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 401261568 bytes 2023-12-12T02:12:31.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 421644288 bytes 2023-12-12T02:12:31.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 464461824 bytes 2023-12-12T02:12:31.902 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 545698816 bytes 2023-12-12T02:12:32.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 586582016 bytes 2023-12-12T02:12:32.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 621148160 bytes 2023-12-12T02:12:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 693971968 bytes 2023-12-12T02:12:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 726340608 bytes 2023-12-12T02:12:32.846 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 746800128 bytes 2023-12-12T02:12:33.060 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 797789184 bytes 2023-12-12T02:12:33.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 817571840 bytes 2023-12-12T02:12:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 831585280 bytes 2023-12-12T02:12:33.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 849390592 bytes 2023-12-12T02:12:33.679 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 892751872 bytes 2023-12-12T02:12:33.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 910028800 bytes 2023-12-12T02:12:34.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 952242176 bytes 2023-12-12T02:12:34.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 995962880 bytes 2023-12-12T02:12:34.374 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1000389632 bytes 2023-12-12T02:12:34.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1014460416 bytes 2023-12-12T02:12:34.643 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1026513920 bytes 2023-12-12T02:12:34.809 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1057076224 bytes 2023-12-12T02:12:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1118457856 bytes 2023-12-12T02:12:35.158 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1139449856 bytes 2023-12-12T02:12:35.370 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1179297792 bytes 2023-12-12T02:12:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1190887424 bytes 2023-12-12T02:12:35.715 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1239213056 bytes 2023-12-12T02:12:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1286758400 bytes 2023-12-12T02:12:36.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1319914496 bytes 2023-12-12T02:12:36.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1358652416 bytes 2023-12-12T02:12:36.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1399258112 bytes 2023-12-12T02:12:36.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1466690560 bytes 2023-12-12T02:12:36.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1488754688 bytes 2023-12-12T02:12:37.056 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1502128128 bytes 2023-12-12T02:12:37.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1582643200 bytes 2023-12-12T02:12:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1607364608 bytes 2023-12-12T02:12:37.665 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1619374080 bytes 2023-12-12T02:12:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1648761856 bytes 2023-12-12T02:12:37.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1649398784 bytes 2023-12-12T02:12:38.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1789619200 bytes 2023-12-12T02:12:38.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1807841280 bytes 2023-12-12T02:12:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1855766528 bytes 2023-12-12T02:12:38.927 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 1902228480 bytes 2023-12-12T02:12:39.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 1973594112 bytes 2023-12-12T02:12:39.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2027116544 bytes 2023-12-12T02:12:39.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2061682688 bytes 2023-12-12T02:12:39.745 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2084301824 bytes 2023-12-12T02:12:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2139996160 bytes 2023-12-12T02:12:40.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2185085952 bytes 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:40.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:40.388 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=10de24b3ff15 2023-12-12T02:12:40.444 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:40.546 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_1 present 10de24b3ff15 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=10de24b3ff15 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:40.547 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=10de24b3ff15 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 10de24b3ff15 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 10de24b3ff15 -o 10de24b3ff15 = 10de24b3ff15 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_1 2023-12-12T02:12:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:40.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2023-12-12T02:12:41.397 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:12:41.395+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f7c14800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=368 2023-12-12T02:12:41.398 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:12:41.395+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f7c14800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=368, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=170, last_copied_object_number=128, snap_seqs={368=18446744073709551614}] 2023-12-12T02:12:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:46.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-12T02:12:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:12:46.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_1 deleted 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_1 2023-12-12T02:12:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:12:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_1 2023-12-12T02:12:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:12:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_1: (2) No such file or directory 2023-12-12T02:12:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:12:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:12:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:12:46.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:12:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_1 2023-12-12T02:12:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_1 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:12:46.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:12:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:46.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:12:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:12:46.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:12:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:12:46.759 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:12:46.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:12:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:12:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:47.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:12:47.822 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:12:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:12:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:12:47.871 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:12:47.874 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:12:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:12:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:49.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:12:49.932 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:12:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:12:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:12:49.979 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:12:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:12:53.984 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:12:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:12:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:12:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:12:53.985 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:12:54.023 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2023-12-12T02:12:54.023 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2023-12-12T02:12:54.023 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2023-12-12T02:12:54.024 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:12:54.022+0000 7f4ceb10e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 60804) UID: 1000 2023-12-12T02:12:54.041 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:12:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:12:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:12:54.090 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:12:54.095 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:13:01.025 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2023-12-12T02:13:01.025 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:13:01.025 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2023-12-12T02:13:01.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2023-12-12T02:13:01.026 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2023-12-12T02:13:01.028 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2023-12-12T02:13:01.029 INFO:tasks.rbd_mirror_thrash:waiting for 102 secs before thrashing 2023-12-12T02:13:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:13:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:02.096 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:13:02.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:13:02.156 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:13:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:02.205 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:02.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:13:03.017 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41442617857494457) > weight (0.25) 2023-12-12T02:13:03.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4782999723508058) > weight (0.25) 2023-12-12T02:13:03.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6048582974791169) > weight (0.25) 2023-12-12T02:13:03.018 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8035303677674319) > weight (0.25) 2023-12-12T02:13:03.018 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2023-12-12T02:13:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_1 2023-12-12T02:13:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:18.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_1 2023-12-12T02:13:18.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2023-12-12T02:13:18.269 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2023-12-12T02:13:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_2 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_2 512M 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:13:18.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_2 2023-12-12T02:13:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_2 2023-12-12T02:13:18.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:13:18.610 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2023-12-12T02:13:19.362 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:13:19.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_2': 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 18c0f2f48ec 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.18c0f2f48ec 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:13:19.420 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:13:18 2023 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:13:18 2023 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:13:18 2023 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: f1e5833c-f5d0-47a8-ac12-8306c83361ad 2023-12-12T02:13:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:13:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_2 2023-12-12T02:13:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:19.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:13:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T02:13:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:13:19.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:13:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 27436032 bytes 2023-12-12T02:13:19.821 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 65167360 bytes 2023-12-12T02:13:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 70424576 bytes 2023-12-12T02:13:20.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 121450496 bytes 2023-12-12T02:13:20.337 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 170306560 bytes 2023-12-12T02:13:20.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 247038976 bytes 2023-12-12T02:13:20.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 268115968 bytes 2023-12-12T02:13:20.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 311814144 bytes 2023-12-12T02:13:21.224 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 391158784 bytes 2023-12-12T02:13:21.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 432756736 bytes 2023-12-12T02:13:21.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 471231488 bytes 2023-12-12T02:13:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 522811392 bytes 2023-12-12T02:13:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 547087360 bytes 2023-12-12T02:13:22.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 608063488 bytes 2023-12-12T02:13:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 640372736 bytes 2023-12-12T02:13:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 695837696 bytes 2023-12-12T02:13:22.766 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 734565376 bytes 2023-12-12T02:13:22.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 770629632 bytes 2023-12-12T02:13:23.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 833869824 bytes 2023-12-12T02:13:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 931077120 bytes 2023-12-12T02:13:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 977871872 bytes 2023-12-12T02:13:23.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1019949056 bytes 2023-12-12T02:13:24.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1060597760 bytes 2023-12-12T02:13:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1107625984 bytes 2023-12-12T02:13:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1133247488 bytes 2023-12-12T02:13:24.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1141164032 bytes 2023-12-12T02:13:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1221485568 bytes 2023-12-12T02:13:25.013 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1262183424 bytes 2023-12-12T02:13:25.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1290799104 bytes 2023-12-12T02:13:25.389 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1345819648 bytes 2023-12-12T02:13:25.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1374731264 bytes 2023-12-12T02:13:25.758 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1417464832 bytes 2023-12-12T02:13:25.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1433988096 bytes 2023-12-12T02:13:26.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1456020480 bytes 2023-12-12T02:13:26.178 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1474498560 bytes 2023-12-12T02:13:26.330 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1497960448 bytes 2023-12-12T02:13:26.481 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1523082240 bytes 2023-12-12T02:13:26.664 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1567099904 bytes 2023-12-12T02:13:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1636825088 bytes 2023-12-12T02:13:27.135 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1684784128 bytes 2023-12-12T02:13:27.440 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1780824064 bytes 2023-12-12T02:13:27.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1817842688 bytes 2023-12-12T02:13:27.729 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1818468352 bytes 2023-12-12T02:13:27.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1843952640 bytes 2023-12-12T02:13:28.120 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1911451648 bytes 2023-12-12T02:13:28.241 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1912085504 bytes 2023-12-12T02:13:28.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1954415616 bytes 2023-12-12T02:13:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2026854400 bytes 2023-12-12T02:13:29.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2067463168 bytes 2023-12-12T02:13:29.280 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2105808896 bytes 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:13:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:13:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:13:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=10de405315e7 2023-12-12T02:13:29.489 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:13:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_2 present 10de405315e7 2023-12-12T02:13:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:29.591 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=10de405315e7 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:13:29.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:13:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=10de405315e7 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 10de405315e7 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 10de405315e7 -o 10de405315e7 = 10de405315e7 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_2 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:29.651 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2023-12-12T02:13:30.332 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:13:30.338+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f7ac3000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=370 2023-12-12T02:13:30.333 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:13:30.338+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f7ac3000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=370, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=172, last_copied_object_number=128, snap_seqs={370=18446744073709551614}] 2023-12-12T02:13:35.332 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:13:35.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:13:35.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:13:35.391 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-12T02:13:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:13:35.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_2 deleted 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_2 2023-12-12T02:13:35.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:13:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_2 2023-12-12T02:13:35.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:13:35.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_2: (2) No such file or directory 2023-12-12T02:13:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:13:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:13:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:13:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:13:35.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_2 2023-12-12T02:13:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.544 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_2 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:13:35.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:13:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:35.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:13:35.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:13:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:35.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:13:36.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:13:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:36.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:13:36.764 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:13:36.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:36.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:36.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:13:38.875 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:13:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:38.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:38.928 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:13:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:13:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:42.929 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:13:42.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:13:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:43.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:43.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:43.037 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:13:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:13:51.038 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:13:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:13:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:13:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:13:51.094 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:13:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:13:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:13:51.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:13:51.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_2 2023-12-12T02:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_2 2023-12-12T02:14:07.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2023-12-12T02:14:07.206 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2023-12-12T02:14:07.498 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:14:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_3 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_3 512M 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_3 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_3 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:14:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2023-12-12T02:14:08.061 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:14:08.074 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2023-12-12T02:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_3': 2023-12-12T02:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:14:08.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 18f68101a64b 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.18f68101a64b 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:14:07 2023 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:14:07 2023 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:14:07 2023 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:14:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 964168a2-1bb0-4c9b-b3cc-6a92651cc3b9 2023-12-12T02:14:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:14:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_3 2023-12-12T02:14:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:08.125 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:14:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=13 2023-12-12T02:14:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:14:08.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:14:08.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 47559680 bytes 2023-12-12T02:14:08.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 79101952 bytes 2023-12-12T02:14:08.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 132295680 bytes 2023-12-12T02:14:08.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 152237056 bytes 2023-12-12T02:14:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 152853504 bytes 2023-12-12T02:14:09.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 153452544 bytes 2023-12-12T02:14:09.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 165623808 bytes 2023-12-12T02:14:09.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 197133312 bytes 2023-12-12T02:14:09.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 213714944 bytes 2023-12-12T02:14:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 219537408 bytes 2023-12-12T02:14:09.870 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 277083136 bytes 2023-12-12T02:14:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 349201408 bytes 2023-12-12T02:14:10.304 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 386904064 bytes 2023-12-12T02:14:10.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 474055680 bytes 2023-12-12T02:14:10.783 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 529759232 bytes 2023-12-12T02:14:11.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 592957440 bytes 2023-12-12T02:14:11.255 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 657928192 bytes 2023-12-12T02:14:11.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 677886976 bytes 2023-12-12T02:14:11.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 719930368 bytes 2023-12-12T02:14:11.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 751592448 bytes 2023-12-12T02:14:12.002 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 812742656 bytes 2023-12-12T02:14:12.019 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-12T02:14:12.019 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-12T02:14:12.019 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2023-12-12T02:14:12.020 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-12T02:14:12.020 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:14:12.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:14:12.025+0000 7fc9b899e640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 64344) UID: 1000 2023-12-12T02:14:12.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 873478144 bytes 2023-12-12T02:14:12.409 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 907142144 bytes 2023-12-12T02:14:12.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 968061952 bytes 2023-12-12T02:14:12.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1000340480 bytes 2023-12-12T02:14:13.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1061816320 bytes 2023-12-12T02:14:13.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1124173824 bytes 2023-12-12T02:14:13.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1315555328 bytes 2023-12-12T02:14:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1402215424 bytes 2023-12-12T02:14:14.225 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1446733824 bytes 2023-12-12T02:14:14.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1494949888 bytes 2023-12-12T02:14:14.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1531063296 bytes 2023-12-12T02:14:14.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1583293440 bytes 2023-12-12T02:14:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1654709248 bytes 2023-12-12T02:14:15.372 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1748739072 bytes 2023-12-12T02:14:15.585 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1814277120 bytes 2023-12-12T02:14:15.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1814880256 bytes 2023-12-12T02:14:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1874880512 bytes 2023-12-12T02:14:16.151 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1888104448 bytes 2023-12-12T02:14:16.303 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1911155712 bytes 2023-12-12T02:14:16.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1997002752 bytes 2023-12-12T02:14:16.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 2017551360 bytes 2023-12-12T02:14:16.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 2055622656 bytes 2023-12-12T02:14:17.097 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 2101577728 bytes 2023-12-12T02:14:17.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2187118592 bytes 2023-12-12T02:14:17.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2232976384 bytes 2023-12-12T02:14:17.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2264032256 bytes 2023-12-12T02:14:17.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2278448128 bytes 2023-12-12T02:14:18.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2351084544 bytes 2023-12-12T02:14:18.122 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-12T02:14:18.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T02:14:18.122 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T02:14:18.125 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T02:14:18.125 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2023-12-12T02:14:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2414955520 bytes 2023-12-12T02:14:18.581 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2449176576 bytes 2023-12-12T02:14:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2493164544 bytes 2023-12-12T02:14:18.981 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2538454016 bytes 2023-12-12T02:14:19.283 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2659775488 bytes 2023-12-12T02:14:19.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2746746880 bytes 2023-12-12T02:14:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2775580672 bytes 2023-12-12T02:14:20.086 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2866402304 bytes 2023-12-12T02:14:20.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2949997568 bytes 2023-12-12T02:14:20.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2982422528 bytes 2023-12-12T02:14:20.695 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 3014790144 bytes 2023-12-12T02:14:20.903 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 3070841856 bytes 2023-12-12T02:14:21.030 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 3093086208 bytes 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:14:21.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:14:21.133 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:14:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=10dedc686033 2023-12-12T02:14:21.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_3 present 10dedc686033 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=10dedc686033 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:21.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:14:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:14:21.351 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=10dedc686033 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 10dedc686033 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 10dedc686033 -o 10dedc686033 = 10dedc686033 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_3 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:21.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2023-12-12T02:14:22.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:14:22.124+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d0fcf860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:14:22.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:14:22.124+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e58280 [3/964168a2-1bb0-4c9b-b3cc-6a92651cc3b9] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:14:22.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:14:22.428+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f77d6c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=372 2023-12-12T02:14:22.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:14:22.428+0000 7f2dedb04640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55d9f77d6c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=372, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=174, last_copied_object_number=128, snap_seqs={372=18446744073709551614}] 2023-12-12T02:14:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:14:27.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:14:27.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-12T02:14:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:14:27.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:14:27.606 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_3 deleted 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_3 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_3 2023-12-12T02:14:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:14:27.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_3: (2) No such file or directory 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_3 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2023-12-12T02:14:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_3 2023-12-12T02:14:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:27.705 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:14:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:27.706 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:27.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:27.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:14:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:14:27.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:14:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:14:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:28.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:28.864 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:14:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:14:28.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:14:28.917 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:14:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:30.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:30.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:14:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:14:31.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:14:31.043 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:14:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:14:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:14:35.150 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:14:35.154 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:14:43.029 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.45346502817087153) > weight (0.25) 2023-12-12T02:14:43.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.67374906390457) > weight (0.25) 2023-12-12T02:14:43.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.42386284734933566) > weight (0.25) 2023-12-12T02:14:43.030 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-12T02:14:43.030 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-12T02:14:43.030 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-12T02:14:43.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:14:43.032+0000 7f2dfb31f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 40080) UID: 1000 2023-12-12T02:14:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:43.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:14:43.262 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:14:43.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:14:49.031 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-12T02:14:49.032 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:14:49.032 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-12T02:14:49.032 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-12T02:14:49.032 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-12T02:14:49.035 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-12T02:14:49.035 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2023-12-12T02:14:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_3 2023-12-12T02:14:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_3 2023-12-12T02:14:59.267 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2023-12-12T02:14:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2023-12-12T02:14:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_4 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_4 512M 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:14:59.615 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_4 2023-12-12T02:14:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_4 2023-12-12T02:14:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:14:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:14:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:14:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:14:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2023-12-12T02:15:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:15:00.385 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_4': 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 193a11a286a 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.193a11a286a 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:15:00.427 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:14:59 2023 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:14:59 2023 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:14:59 2023 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 45804996-9ab8-43c0-977c-864980991abf 2023-12-12T02:15:00.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_4 2023-12-12T02:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:00.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:15:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:15:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:15:00.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:15:00.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 49321984 bytes 2023-12-12T02:15:00.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 122722304 bytes 2023-12-12T02:15:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 152103936 bytes 2023-12-12T02:15:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 228706304 bytes 2023-12-12T02:15:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 289224704 bytes 2023-12-12T02:15:01.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 350804992 bytes 2023-12-12T02:15:02.031 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 442254336 bytes 2023-12-12T02:15:02.223 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 491834368 bytes 2023-12-12T02:15:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 526098432 bytes 2023-12-12T02:15:02.613 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 561261568 bytes 2023-12-12T02:15:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 635320320 bytes 2023-12-12T02:15:02.990 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 645395456 bytes 2023-12-12T02:15:03.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 680434688 bytes 2023-12-12T02:15:03.281 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 681052160 bytes 2023-12-12T02:15:03.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 707107840 bytes 2023-12-12T02:15:03.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 711569408 bytes 2023-12-12T02:15:03.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 745030656 bytes 2023-12-12T02:15:03.955 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 798310400 bytes 2023-12-12T02:15:04.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 839841792 bytes 2023-12-12T02:15:04.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 908658688 bytes 2023-12-12T02:15:04.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 942563328 bytes 2023-12-12T02:15:04.691 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 943182848 bytes 2023-12-12T02:15:04.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 944534528 bytes 2023-12-12T02:15:05.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1031877632 bytes 2023-12-12T02:15:05.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1094350848 bytes 2023-12-12T02:15:05.655 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1190104064 bytes 2023-12-12T02:15:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1201750016 bytes 2023-12-12T02:15:05.977 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1249389568 bytes 2023-12-12T02:15:06.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1273666560 bytes 2023-12-12T02:15:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:15:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:15:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 present 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:06.620 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:15:06.624 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:15:07.625 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 present 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:07.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:15:07.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:15:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 present 2023-12-12T02:15:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:09.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:09.684 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:09.734 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:15:09.735 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:15:13.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 present 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:13.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:13.738 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:13.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:13.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:15:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:15:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 present 2023-12-12T02:15:21.790 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=15b68bd2520f 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 15b68bd2520f 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z -o = 15b68bd2520f 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_4 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2023-12-12T02:15:22.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:15:22.123+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d0f67a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:15:22.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:15:22.123+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0e58280 [3/45804996-9ab8-43c0-977c-864980991abf] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:15:22.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:15:22.395+0000 7f91b5a0c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de53cbbc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=374 2023-12-12T02:15:22.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:15:22.395+0000 7f91b5a0c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de53cbbc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=374, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=176, last_copied_object_number=128, snap_seqs={374=18446744073709551614}] 2023-12-12T02:15:27.402 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:27.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:27.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:15:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_4 deleted 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:27.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_4 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_4 2023-12-12T02:15:27.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_4: (2) No such file or directory 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_4 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_4 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:15:27.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:15:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:27.773 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:15:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:15:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:28.774 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:28.775 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:15:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:15:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:28.881 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:28.885 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:15:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:15:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:30.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:30.887 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:15:30.944 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:15:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:30.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:30.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:15:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:15:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:34.998 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:15:35.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:15:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:35.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:35.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:15:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_4 2023-12-12T02:15:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_4 2023-12-12T02:15:43.108 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2023-12-12T02:15:43.165 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2023-12-12T02:15:43.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:15:43.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_5 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_5 512M 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:15:43.464 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_5 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_5 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:15:43.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2023-12-12T02:15:44.399 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:15:44.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_5': 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 196a561fd095 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.196a561fd095 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:15:43 2023 2023-12-12T02:15:44.456 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:15:43 2023 2023-12-12T02:15:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:15:43 2023 2023-12-12T02:15:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:15:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:15:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: c979b9d6-666a-4cab-9ed6-56f01c2c01b1 2023-12-12T02:15:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:15:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_5 2023-12-12T02:15:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:44.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:44.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:15:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=5 2023-12-12T02:15:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:15:44.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:15:44.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 72663040 bytes 2023-12-12T02:15:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 129554432 bytes 2023-12-12T02:15:45.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 190189568 bytes 2023-12-12T02:15:45.293 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 196128768 bytes 2023-12-12T02:15:45.614 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 292156416 bytes 2023-12-12T02:15:45.754 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 309987328 bytes 2023-12-12T02:15:45.915 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 342141952 bytes 2023-12-12T02:15:46.069 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 366086144 bytes 2023-12-12T02:15:46.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 400264192 bytes 2023-12-12T02:15:46.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 450764800 bytes 2023-12-12T02:15:46.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 488331264 bytes 2023-12-12T02:15:46.886 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 552581120 bytes 2023-12-12T02:15:47.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 663041024 bytes 2023-12-12T02:15:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 752196608 bytes 2023-12-12T02:15:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 849786880 bytes 2023-12-12T02:15:47.973 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 887135232 bytes 2023-12-12T02:15:48.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 930608128 bytes 2023-12-12T02:15:48.377 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 975469568 bytes 2023-12-12T02:15:48.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 1004993536 bytes 2023-12-12T02:15:48.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 1048565760 bytes 2023-12-12T02:15:48.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 1076708352 bytes 2023-12-12T02:15:49.100 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1128693760 bytes 2023-12-12T02:15:49.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1139594240 bytes 2023-12-12T02:15:49.416 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1176013824 bytes 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:49.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:15:49.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=15b6724d1be5 2023-12-12T02:15:49.516 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:15:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_5 present 15b6724d1be5 2023-12-12T02:15:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:49.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=15b6724d1be5 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:15:49.618 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=15b6724d1be5 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 15b6724d1be5 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 15b6724d1be5 -o 15b6724d1be5 = 15b6724d1be5 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_5 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:49.677 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2023-12-12T02:15:50.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:15:50.443+0000 7f91b5a0c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de52ead400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=376 2023-12-12T02:15:50.442 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:15:50.443+0000 7f91b5a0c640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55de52ead400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=376, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=178, last_copied_object_number=128, snap_seqs={376=18446744073709551614}] 2023-12-12T02:15:52.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:15:52.123+0000 7fde2dfeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d0ee1380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:15:52.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:15:52.123+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777900 [3/c979b9d6-666a-4cab-9ed6-56f01c2c01b1] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:15:55.434 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:15:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:15:55.450 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:55.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-12T02:15:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:15:55.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_5 deleted 2023-12-12T02:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.607 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_5 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.608 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_5 2023-12-12T02:15:55.609 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:15:55.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_5: (2) No such file or directory 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_5 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.657 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2023-12-12T02:15:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_5 2023-12-12T02:15:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.708 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:15:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:15:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:55.709 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:15:55.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:15:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:55.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:55.805 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:15:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:15:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:56.806 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:15:57.078 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:57.132 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:57.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:15:58.125 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-12T02:15:58.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-12T02:15:58.126 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2023-12-12T02:15:58.127 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:15:58.127+0000 7fb12b1a7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 73506) UID: 1000 2023-12-12T02:15:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:15:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:15:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:15:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:15:59.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:15:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:15:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:15:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:15:59.243 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:15:59.247 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:16:03.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:16:03.306 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:16:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:16:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:16:03.355 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:16:03.358 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:16:06.127 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-12T02:16:06.127 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:16:06.128 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-12T02:16:06.128 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T02:16:06.128 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T02:16:06.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T02:16:06.131 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-12T02:16:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:16:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:16:11.360 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:16:11.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:16:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:16:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:16:11.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:16:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:16:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_5 2023-12-12T02:16:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_5 2023-12-12T02:16:27.468 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2023-12-12T02:16:27.524 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2023-12-12T02:16:27.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_6 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_6 512M 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:16:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:16:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_6 2023-12-12T02:16:27.850 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_6 2023-12-12T02:16:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:16:27.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2023-12-12T02:16:28.036 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8524185960507082) > weight (0.25) 2023-12-12T02:16:28.036 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5926915719833148) > weight (0.25) 2023-12-12T02:16:28.036 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2023-12-12T02:16:28.037 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2023-12-12T02:16:28.037 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-12T02:16:28.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:16:28.042+0000 7f91c3a28640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 64158) UID: 1000 2023-12-12T02:16:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:16:28.164 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_6': 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 19a9b6e22c21 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.19a9b6e22c21 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:16:27 2023 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:16:27 2023 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:16:27 2023 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:16:28.208 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:16:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 893fe4ed-6f17-4d25-9ef9-ced239deb527 2023-12-12T02:16:28.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:16:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_6 2023-12-12T02:16:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:16:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:28.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:16:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=6 2023-12-12T02:16:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:16:28.213 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:16:28.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 14445568 bytes 2023-12-12T02:16:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 16412672 bytes 2023-12-12T02:16:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 66113536 bytes 2023-12-12T02:16:28.854 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 88758272 bytes 2023-12-12T02:16:29.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 162098176 bytes 2023-12-12T02:16:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 174140416 bytes 2023-12-12T02:16:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 236020736 bytes 2023-12-12T02:16:29.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 282235904 bytes 2023-12-12T02:16:29.966 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 373311488 bytes 2023-12-12T02:16:30.037 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2023-12-12T02:16:30.038 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:16:30.038 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2023-12-12T02:16:30.038 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2023-12-12T02:16:30.038 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2023-12-12T02:16:30.040 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2023-12-12T02:16:30.041 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2023-12-12T02:16:30.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 406401024 bytes 2023-12-12T02:16:30.346 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 450563072 bytes 2023-12-12T02:16:30.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 481737728 bytes 2023-12-12T02:16:30.712 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 520833024 bytes 2023-12-12T02:16:31.000 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 589127680 bytes 2023-12-12T02:16:31.181 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 619914240 bytes 2023-12-12T02:16:31.335 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 640398336 bytes 2023-12-12T02:16:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 711867392 bytes 2023-12-12T02:16:31.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 735771648 bytes 2023-12-12T02:16:31.904 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 770489344 bytes 2023-12-12T02:16:32.044 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 801147904 bytes 2023-12-12T02:16:32.338 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 874769408 bytes 2023-12-12T02:16:32.462 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 888586240 bytes 2023-12-12T02:16:32.569 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 889181184 bytes 2023-12-12T02:16:32.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 889805824 bytes 2023-12-12T02:16:32.816 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 918986752 bytes 2023-12-12T02:16:32.962 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 926119936 bytes 2023-12-12T02:16:33.196 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 987539456 bytes 2023-12-12T02:16:33.382 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1041618944 bytes 2023-12-12T02:16:33.593 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1088848896 bytes 2023-12-12T02:16:33.798 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1138940928 bytes 2023-12-12T02:16:33.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1176631296 bytes 2023-12-12T02:16:34.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1191781376 bytes 2023-12-12T02:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:16:34.220 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:34.221 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:16:34.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:16:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:34.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:34.367 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:34.415 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:16:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:35.417 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:35.418 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:35.463 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:35.467 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:37.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:37.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:37.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:37.519 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:41.521 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:41.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:41.570 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:41.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:16:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:49.577 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:49.623 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:49.627 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:16:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:16:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:57.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:16:57.630 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:16:57.674 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:16:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:16:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:16:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = deleted 2023-12-12T02:16:57.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:17:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 present 2023-12-12T02:17:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:05.680 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:17:05.681 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=162b5e2ef7e 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 162b5e2ef7e 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z -o = 162b5e2ef7e 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_6 2023-12-12T02:17:05.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:05.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2023-12-12T02:17:06.437 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.470 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:17:06.552 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:17:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:17:06.557 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_6 deleted 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_6 2023-12-12T02:17:06.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_6 2023-12-12T02:17:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_6: (2) No such file or directory 2023-12-12T02:17:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:17:06.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_6 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2023-12-12T02:17:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_6 2023-12-12T02:17:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:06.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_6 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_6 2023-12-12T02:17:06.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2023-12-12T02:17:06.795 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2023-12-12T02:17:07.085 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_7 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_7 512M 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:17:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_7 2023-12-12T02:17:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_7 2023-12-12T02:17:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:17:07.140 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2023-12-12T02:17:08.192 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:17:08.204 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2023-12-12T02:17:08.252 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_7': 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 19c2a0b6385c 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.19c2a0b6385c 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:17:07 2023 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:17:07 2023 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:17:07 2023 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:17:08.253 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:17:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: e00c5e3f-8c81-4b48-9558-6035cb769d18 2023-12-12T02:17:08.254 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_7 2023-12-12T02:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:08.257 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:17:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=14 2023-12-12T02:17:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:17:08.258 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:17:08.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 82972672 bytes 2023-12-12T02:17:08.890 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 197303296 bytes 2023-12-12T02:17:09.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 224492544 bytes 2023-12-12T02:17:09.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 271165440 bytes 2023-12-12T02:17:09.473 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 353864704 bytes 2023-12-12T02:17:09.703 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 409535488 bytes 2023-12-12T02:17:09.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 449384448 bytes 2023-12-12T02:17:10.114 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 499553280 bytes 2023-12-12T02:17:10.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 533518336 bytes 2023-12-12T02:17:10.503 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 578767872 bytes 2023-12-12T02:17:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 626085888 bytes 2023-12-12T02:17:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 682041344 bytes 2023-12-12T02:17:11.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 703933440 bytes 2023-12-12T02:17:11.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 727439360 bytes 2023-12-12T02:17:11.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 741737472 bytes 2023-12-12T02:17:11.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 787770368 bytes 2023-12-12T02:17:11.736 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 825210880 bytes 2023-12-12T02:17:11.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 860736512 bytes 2023-12-12T02:17:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 899428352 bytes 2023-12-12T02:17:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 964854784 bytes 2023-12-12T02:17:12.514 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 996733952 bytes 2023-12-12T02:17:12.629 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 998315008 bytes 2023-12-12T02:17:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1004983296 bytes 2023-12-12T02:17:12.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1009569792 bytes 2023-12-12T02:17:13.091 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1063511040 bytes 2023-12-12T02:17:13.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1084449792 bytes 2023-12-12T02:17:13.411 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1108965376 bytes 2023-12-12T02:17:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1126048768 bytes 2023-12-12T02:17:13.666 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1132007424 bytes 2023-12-12T02:17:13.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1170917376 bytes 2023-12-12T02:17:14.032 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1207777280 bytes 2023-12-12T02:17:14.142 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1208393728 bytes 2023-12-12T02:17:14.350 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1254026240 bytes 2023-12-12T02:17:14.526 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1286030336 bytes 2023-12-12T02:17:14.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1366646784 bytes 2023-12-12T02:17:15.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1480643584 bytes 2023-12-12T02:17:15.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1541625856 bytes 2023-12-12T02:17:15.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1548721152 bytes 2023-12-12T02:17:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1625272320 bytes 2023-12-12T02:17:15.923 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1673620480 bytes 2023-12-12T02:17:16.055 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1674223616 bytes 2023-12-12T02:17:16.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1716784128 bytes 2023-12-12T02:17:16.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1780836352 bytes 2023-12-12T02:17:16.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1793576960 bytes 2023-12-12T02:17:16.787 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1820426240 bytes 2023-12-12T02:17:16.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1838332928 bytes 2023-12-12T02:17:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 1870172160 bytes 2023-12-12T02:17:17.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 1943300096 bytes 2023-12-12T02:17:17.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 1991614464 bytes 2023-12-12T02:17:17.700 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:17:17.700+0000 7f5c5553d640 -1 bdev(0x558c1c318e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c5553d640 time 2023-12-12T02:17:17.706175+0000 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x558c18a09750] 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:*** Caught signal (Aborted) ** 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: in thread 7f5c5553d640 thread_name:bstore_aio 2023-12-12T02:17:17.705 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:17:17.704+0000 7f5c4dd2e640 -1 bdev(0x558c1c319180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:17:17.704+0000 7f5c5553d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c5553d640 time 2023-12-12T02:17:17.706175+0000 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x558c18a09750] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5c63335520] 2023-12-12T02:17:17.706 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: pthread_kill() 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: raise() 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: abort() 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x558c18a09805] 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 7: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:2023-12-12T02:17:17.708+0000 7f5c5553d640 -1 *** Caught signal (Aborted) ** 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: in thread 7f5c5553d640 thread_name:bstore_aio 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.707 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5c63335520] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: pthread_kill() 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: raise() 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: abort() 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x558c18a09805] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 7: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-12T02:17:17.708 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -588> 2023-12-12T02:17:17.700+0000 7f5c5553d640 -1 bdev(0x558c1c318e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -477> 2023-12-12T02:17:17.704+0000 7f5c4dd2e640 -1 bdev(0x558c1c319180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -437> 2023-12-12T02:17:17.704+0000 7f5c5553d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c5553d640 time 2023-12-12T02:17:17.706175+0000 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x558c18a09750] 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.720 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -17> 2023-12-12T02:17:17.708+0000 7f5c5553d640 -1 *** Caught signal (Aborted) ** 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: in thread 7f5c5553d640 thread_name:bstore_aio 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5c63335520] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: pthread_kill() 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: raise() 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: abort() 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x558c18a09805] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 7: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-12T02:17:17.721 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -967> 2023-12-12T02:17:17.700+0000 7f5c5553d640 -1 bdev(0x558c1c318e00 /var/lib/ceph/osd/cluster2-0/block) _aio_thread got r=-28 ((28) No space left on device) 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -856> 2023-12-12T02:17:17.704+0000 7f5c4dd2e640 -1 bdev(0x558c1c319180 /var/lib/ceph/osd/cluster2-0/block) _sync_write pwritev error: (28) No space left on device 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -816> 2023-12-12T02:17:17.704+0000 7f5c5553d640 -1 ./src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f5c5553d640 time 2023-12-12T02:17:17.706175+0000 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:./src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xc6) [0x558c18a09750] 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.729 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: -396> 2023-12-12T02:17:17.708+0000 7f5c5553d640 -1 *** Caught signal (Aborted) ** 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: in thread 7f5c5553d640 thread_name:bstore_aio 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev) 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 1: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5c63335520] 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2: pthread_kill() 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 3: raise() 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 4: abort() 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x17b) [0x558c18a09805] 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 6: (KernelDevice::_aio_thread()+0x9bc) [0x558c192f17fc] 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 7: ceph-osd(+0x15b40e1) [0x558c192d40e1] 2023-12-12T02:17:17.730 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 8: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5c63387b43] 2023-12-12T02:17:17.731 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 9: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5c63419a00] 2023-12-12T02:17:17.731 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-12-12T02:17:17.731 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr: 2023-12-12T02:17:17.817 INFO:tasks.ceph.cluster2.osd.0.smithi140.stderr:daemon-helper: command failed with exit status 5 2023-12-12T02:17:18.543 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2023-12-12T02:17:19.132 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2023-12-12T02:17:19.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2023-12-12T02:17:19.133 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2023-12-12T02:17:19.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:17:19.136+0000 7f0d3cf79640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 75076) UID: 1000 2023-12-12T02:17:19.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2028153856 bytes 2023-12-12T02:17:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2042875904 bytes 2023-12-12T02:17:19.740 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2049091584 bytes 2023-12-12T02:17:19.898 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2098685952 bytes 2023-12-12T02:17:20.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2152596480 bytes 2023-12-12T02:17:20.134 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2023-12-12T02:17:20.134 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:17:20.134 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2023-12-12T02:17:20.134 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2023-12-12T02:17:20.135 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2023-12-12T02:17:20.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2023-12-12T02:17:20.137 INFO:tasks.rbd_mirror_thrash:waiting for 79 secs before thrashing 2023-12-12T02:17:20.144 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2163943424 bytes 2023-12-12T02:17:20.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2218752000 bytes 2023-12-12T02:17:20.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2272516096 bytes 2023-12-12T02:17:20.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2281811968 bytes 2023-12-12T02:17:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2358964224 bytes 2023-12-12T02:17:20.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2429786112 bytes 2023-12-12T02:17:21.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2435024896 bytes 2023-12-12T02:17:21.124 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2457212928 bytes 2023-12-12T02:17:21.278 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2527861760 bytes 2023-12-12T02:17:21.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2546983936 bytes 2023-12-12T02:17:21.511 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2599370752 bytes 2023-12-12T02:17:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2669893632 bytes 2023-12-12T02:17:21.865 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2720603136 bytes 2023-12-12T02:17:22.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2780419072 bytes 2023-12-12T02:17:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2843731968 bytes 2023-12-12T02:17:22.263 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:17:22.264 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:17:22.265 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=162b7be8ba53 2023-12-12T02:17:22.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_7 present 162b7be8ba53 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=162b7be8ba53 2023-12-12T02:17:22.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:17:22.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=162b7be8ba53 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 162b7be8ba53 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 162b7be8ba53 -o 162b7be8ba53 = 162b7be8ba53 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_7 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:22.492 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2023-12-12T02:17:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:17:23.508+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646359000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=380 2023-12-12T02:17:23.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:17:23.508+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646359000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=380, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=17c, last_copied_object_number=128, snap_seqs={380=18446744073709551614}] 2023-12-12T02:17:23.946 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~5s 2023-12-12T02:17:28.517 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:17:28.530 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:17:28.531 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:28.579 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-12T02:17:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:17:28.584 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_7 deleted 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:17:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_7 2023-12-12T02:17:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:17:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_7 2023-12-12T02:17:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:17:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_7: (2) No such file or directory 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_7 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.737 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2023-12-12T02:17:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_7 2023-12-12T02:17:28.784 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:28.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:17:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:17:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:17:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:17:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:17:28.882 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:17:29.350 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~11s 2023-12-12T02:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:29.884 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:17:29.943 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:17:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:17:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:17:29.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:17:29.995 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:17:32.052 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:17:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:17:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:17:32.099 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:17:32.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:17:34.754 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~16s 2023-12-12T02:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:36.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:17:36.162 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:17:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:17:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:17:36.209 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:17:36.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:17:40.158 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~22s 2023-12-12T02:17:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:17:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:17:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:17:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:17:44.214 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:17:44.271 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:17:44.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:17:44.321 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:17:45.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~27s 2023-12-12T02:17:50.966 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~32s 2023-12-12T02:17:56.369 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~38s 2023-12-12T02:17:59.041 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5437229039916601) > weight (0.25) 2023-12-12T02:17:59.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6922347079460025) > weight (0.25) 2023-12-12T02:17:59.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4512922202523795) > weight (0.25) 2023-12-12T02:17:59.042 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6370940213716833) > weight (0.25) 2023-12-12T02:17:59.042 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2023-12-12T02:18:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_7 2023-12-12T02:18:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_7 2023-12-12T02:18:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2023-12-12T02:18:00.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2023-12-12T02:18:00.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_8 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_8 512M 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:00.602 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_8 2023-12-12T02:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_8 2023-12-12T02:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:00.652 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:18:00.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2023-12-12T02:18:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:18:01.480 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2023-12-12T02:18:01.527 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_8': 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 19f579090fc6 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.19f579090fc6 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:18:00 2023 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:18:00 2023 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:18:00 2023 2023-12-12T02:18:01.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:18:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:18:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 4c94d724-10f8-4be4-abd5-ab16e0de9fbe 2023-12-12T02:18:01.529 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_8 2023-12-12T02:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:01.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=11 2023-12-12T02:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:18:01.533 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:18:01.739 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 51641344 bytes 2023-12-12T02:18:01.773 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~43s 2023-12-12T02:18:01.829 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 68466688 bytes 2023-12-12T02:18:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 118501376 bytes 2023-12-12T02:18:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 143401984 bytes 2023-12-12T02:18:02.185 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 189812736 bytes 2023-12-12T02:18:02.365 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 260150272 bytes 2023-12-12T02:18:02.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 295111680 bytes 2023-12-12T02:18:02.617 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 348726272 bytes 2023-12-12T02:18:02.749 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 371870720 bytes 2023-12-12T02:18:02.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 395659264 bytes 2023-12-12T02:18:02.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 428771328 bytes 2023-12-12T02:18:03.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 518600704 bytes 2023-12-12T02:18:03.345 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 610518016 bytes 2023-12-12T02:18:03.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 635791360 bytes 2023-12-12T02:18:03.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 636398592 bytes 2023-12-12T02:18:03.696 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 690956288 bytes 2023-12-12T02:18:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 705583104 bytes 2023-12-12T02:18:03.961 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 775499776 bytes 2023-12-12T02:18:04.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 796049408 bytes 2023-12-12T02:18:04.188 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 824429568 bytes 2023-12-12T02:18:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 841235456 bytes 2023-12-12T02:18:04.433 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 886930432 bytes 2023-12-12T02:18:04.553 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 915659776 bytes 2023-12-12T02:18:04.689 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 950413312 bytes 2023-12-12T02:18:04.853 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1015799808 bytes 2023-12-12T02:18:04.940 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1016980480 bytes 2023-12-12T02:18:05.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1052948480 bytes 2023-12-12T02:18:05.194 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1100265472 bytes 2023-12-12T02:18:05.381 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1198418944 bytes 2023-12-12T02:18:05.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1272336384 bytes 2023-12-12T02:18:05.682 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1284773888 bytes 2023-12-12T02:18:05.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1316649984 bytes 2023-12-12T02:18:05.954 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1387820032 bytes 2023-12-12T02:18:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1425372160 bytes 2023-12-12T02:18:06.248 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1482397696 bytes 2023-12-12T02:18:06.390 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1541098496 bytes 2023-12-12T02:18:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1570540544 bytes 2023-12-12T02:18:06.667 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1615718400 bytes 2023-12-12T02:18:06.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1697524736 bytes 2023-12-12T02:18:06.963 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1723586560 bytes 2023-12-12T02:18:07.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1773196288 bytes 2023-12-12T02:18:07.180 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~49s 2023-12-12T02:18:07.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1844926464 bytes 2023-12-12T02:18:07.432 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1876186112 bytes 2023-12-12T02:18:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1899960320 bytes 2023-12-12T02:18:07.723 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1974640640 bytes 2023-12-12T02:18:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2046180352 bytes 2023-12-12T02:18:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2072794112 bytes 2023-12-12T02:18:08.222 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2165726208 bytes 2023-12-12T02:18:08.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2223559680 bytes 2023-12-12T02:18:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2275773440 bytes 2023-12-12T02:18:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2318497792 bytes 2023-12-12T02:18:08.808 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2385977344 bytes 2023-12-12T02:18:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2403449856 bytes 2023-12-12T02:18:09.087 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2460713984 bytes 2023-12-12T02:18:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2482577408 bytes 2023-12-12T02:18:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2508924928 bytes 2023-12-12T02:18:09.423 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2542215168 bytes 2023-12-12T02:18:09.621 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2614104064 bytes 2023-12-12T02:18:09.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2636539904 bytes 2023-12-12T02:18:09.837 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2660687872 bytes 2023-12-12T02:18:09.937 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2701421568 bytes 2023-12-12T02:18:10.071 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2724988928 bytes 2023-12-12T02:18:10.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2793577472 bytes 2023-12-12T02:18:10.366 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2840351744 bytes 2023-12-12T02:18:10.487 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2870987776 bytes 2023-12-12T02:18:10.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2882241536 bytes 2023-12-12T02:18:10.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2886996992 bytes 2023-12-12T02:18:10.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2959626240 bytes 2023-12-12T02:18:11.046 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 3057634304 bytes 2023-12-12T02:18:11.168 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 3086734336 bytes 2023-12-12T02:18:11.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 3125859328 bytes 2023-12-12T02:18:11.395 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3136277504 bytes 2023-12-12T02:18:11.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3162108928 bytes 2023-12-12T02:18:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3200141312 bytes 2023-12-12T02:18:11.815 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7500 IOs, 3263729664 bytes 2023-12-12T02:18:11.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7600 IOs, 3273610240 bytes 2023-12-12T02:18:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7700 IOs, 3304491008 bytes 2023-12-12T02:18:12.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7800 IOs, 3335150592 bytes 2023-12-12T02:18:12.323 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7900 IOs, 3401181184 bytes 2023-12-12T02:18:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:8000 IOs, 3405067264 bytes 2023-12-12T02:18:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:12.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:18:12.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:18:12.584 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~54s 2023-12-12T02:18:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=162b8dabb829 2023-12-12T02:18:12.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_8 present 162b8dabb829 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=162b8dabb829 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:12.697 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:18:12.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=162b8dabb829 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 162b8dabb829 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 162b8dabb829 -o 162b8dabb829 = 162b8dabb829 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:18:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:18:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_8 2023-12-12T02:18:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2023-12-12T02:18:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:18:13.446+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646788800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=382 2023-12-12T02:18:13.443 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:18:13.446+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646788800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=382, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=17e, last_copied_object_number=128, snap_seqs={382=18446744073709551614}] 2023-12-12T02:18:17.988 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~59s 2023-12-12T02:18:18.445 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:18:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2023-12-12T02:18:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.459 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:18:18.461 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:18:18.505 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-12T02:18:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:18:18.509 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_8 deleted 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:18:18.611 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_8 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_8 2023-12-12T02:18:18.612 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:18:18.659 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_8: (2) No such file or directory 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_8 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2023-12-12T02:18:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_8 2023-12-12T02:18:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.713 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:18:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:18.714 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:18.762 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:18:18.807 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:18:18.810 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:18:19.811 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:19.812 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:18:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:18:19.914 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:18:19.918 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:18:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:21.976 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:18:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:18:22.022 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:18:22.025 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:18:23.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~65s 2023-12-12T02:18:26.026 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:26.027 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:26.083 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:18:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:18:26.131 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:18:26.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:18:28.796 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~70s 2023-12-12T02:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:34.136 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:34.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:34.200 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~76s 2023-12-12T02:18:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:18:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:18:34.239 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:18:34.242 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:18:39.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7135186982779499) > weight (0.25) 2023-12-12T02:18:39.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8475749444903715) > weight (0.25) 2023-12-12T02:18:39.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.888326595062531) > weight (0.25) 2023-12-12T02:18:39.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5583345441611633) > weight (0.25) 2023-12-12T02:18:39.138 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-12T02:18:39.604 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~81s 2023-12-12T02:18:45.008 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~86s 2023-12-12T02:18:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_8 2023-12-12T02:18:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:50.244 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_8 2023-12-12T02:18:50.245 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2023-12-12T02:18:50.301 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2023-12-12T02:18:50.412 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~92s 2023-12-12T02:18:50.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_9 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_9 512M 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:18:50.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:18:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:18:50.537 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_9 2023-12-12T02:18:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_9 2023-12-12T02:18:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:50.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:18:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:18:50.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2023-12-12T02:18:51.500 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:18:51.512 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_9': 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1a1ec42274de 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1a1ec42274de 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:18:50 2023 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:18:50 2023 2023-12-12T02:18:51.560 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:18:50 2023 2023-12-12T02:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 8e7177b5-b5e7-4bfa-bf0c-c5c6e431f037 2023-12-12T02:18:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_9 2023-12-12T02:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:18:51.564 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:18:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T02:18:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:18:51.565 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:18:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 34779136 bytes 2023-12-12T02:18:51.876 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 103499776 bytes 2023-12-12T02:18:51.992 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 140273664 bytes 2023-12-12T02:18:52.115 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 185574400 bytes 2023-12-12T02:18:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 204191744 bytes 2023-12-12T02:18:52.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 241619968 bytes 2023-12-12T02:18:52.504 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 293298176 bytes 2023-12-12T02:18:52.671 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 366941184 bytes 2023-12-12T02:18:52.851 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 442803200 bytes 2023-12-12T02:18:52.939 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 454156288 bytes 2023-12-12T02:18:53.110 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 500067328 bytes 2023-12-12T02:18:53.229 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 527558656 bytes 2023-12-12T02:18:53.393 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 598185984 bytes 2023-12-12T02:18:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 676370432 bytes 2023-12-12T02:18:53.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 701232128 bytes 2023-12-12T02:18:53.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 738243584 bytes 2023-12-12T02:18:53.916 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 763757568 bytes 2023-12-12T02:18:54.017 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 777960448 bytes 2023-12-12T02:18:54.130 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 819771392 bytes 2023-12-12T02:18:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 901114880 bytes 2023-12-12T02:18:54.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 936051712 bytes 2023-12-12T02:18:54.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 1008944128 bytes 2023-12-12T02:18:54.768 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 1068541952 bytes 2023-12-12T02:18:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 1138516992 bytes 2023-12-12T02:18:55.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1152405504 bytes 2023-12-12T02:18:55.134 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1198701568 bytes 2023-12-12T02:18:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1251180544 bytes 2023-12-12T02:18:55.451 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1287179264 bytes 2023-12-12T02:18:55.556 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1314093056 bytes 2023-12-12T02:18:55.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1347752960 bytes 2023-12-12T02:18:55.799 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1394093056 bytes 2023-12-12T02:18:55.815 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~97s 2023-12-12T02:18:55.925 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1417991168 bytes 2023-12-12T02:18:56.033 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1442927616 bytes 2023-12-12T02:18:56.119 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1456269312 bytes 2023-12-12T02:18:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1531385856 bytes 2023-12-12T02:18:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1598126080 bytes 2023-12-12T02:18:56.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1661097984 bytes 2023-12-12T02:18:56.797 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1716955136 bytes 2023-12-12T02:18:56.910 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1754316800 bytes 2023-12-12T02:18:57.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1771272192 bytes 2023-12-12T02:18:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1854471168 bytes 2023-12-12T02:18:57.292 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1879420928 bytes 2023-12-12T02:18:57.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1923104768 bytes 2023-12-12T02:18:57.596 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1965622272 bytes 2023-12-12T02:18:57.724 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 2011342848 bytes 2023-12-12T02:18:57.803 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 2013323264 bytes 2023-12-12T02:18:57.945 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2062686208 bytes 2023-12-12T02:18:58.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2144831488 bytes 2023-12-12T02:18:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2197124096 bytes 2023-12-12T02:18:58.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2231308288 bytes 2023-12-12T02:18:58.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2231905280 bytes 2023-12-12T02:18:58.644 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2259474432 bytes 2023-12-12T02:18:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2274356224 bytes 2023-12-12T02:18:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2292360192 bytes 2023-12-12T02:18:58.993 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2365217792 bytes 2023-12-12T02:18:59.121 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2373946368 bytes 2023-12-12T02:18:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2378280960 bytes 2023-12-12T02:18:59.401 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2450830336 bytes 2023-12-12T02:18:59.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2489314304 bytes 2023-12-12T02:18:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2555578368 bytes 2023-12-12T02:18:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2587355136 bytes 2023-12-12T02:18:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2651523072 bytes 2023-12-12T02:19:00.170 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2709998592 bytes 2023-12-12T02:19:00.288 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2731941888 bytes 2023-12-12T02:19:00.403 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2760374272 bytes 2023-12-12T02:19:00.528 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 2783071232 bytes 2023-12-12T02:19:00.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 2824034304 bytes 2023-12-12T02:19:00.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 2848388096 bytes 2023-12-12T02:19:00.891 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 2872608768 bytes 2023-12-12T02:19:00.975 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 2885168128 bytes 2023-12-12T02:19:01.077 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 2907686912 bytes 2023-12-12T02:19:01.218 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~103s 2023-12-12T02:19:01.317 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7200 IOs, 3031892992 bytes 2023-12-12T02:19:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7300 IOs, 3044293632 bytes 2023-12-12T02:19:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7400 IOs, 3086033920 bytes 2023-12-12T02:19:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:01.574 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:19:01.575 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=162bb65f3ac6 2023-12-12T02:19:01.628 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_9 present 162bb65f3ac6 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=162bb65f3ac6 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:19:01.730 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:19:01.731 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=162bb65f3ac6 2023-12-12T02:19:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 162bb65f3ac6 2023-12-12T02:19:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 162bb65f3ac6 -o 162bb65f3ac6 = 162bb65f3ac6 2023-12-12T02:19:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:19:01.788 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_9 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2023-12-12T02:19:02.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:19:02.525+0000 7f4138ee6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646359400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=384 2023-12-12T02:19:02.523 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:19:02.525+0000 7f4138ee6640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646359400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=384, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=180, last_copied_object_number=128, snap_seqs={384=18446744073709551614}] 2023-12-12T02:19:03.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5388099577046124) > weight (0.25) 2023-12-12T02:19:03.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5782107331509974) > weight (0.25) 2023-12-12T02:19:03.043 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8349784050542273) > weight (0.25) 2023-12-12T02:19:03.043 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2023-12-12T02:19:03.044 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2023-12-12T02:19:03.044 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2023-12-12T02:19:03.045 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:19:03.045+0000 7f561c1b9640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 73984) UID: 1000 2023-12-12T02:19:04.209 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:19:04.208+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1cfb69ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:19:04.210 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:19:04.208+0000 7fde347f8640 -1 rbd::mirror::ImageReplayer: 0x55a1d0776500 [3/8e7177b5-b5e7-4bfa-bf0c-c5c6e431f037] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:19:06.045 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2023-12-12T02:19:06.045 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:19:06.046 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2023-12-12T02:19:06.046 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2023-12-12T02:19:06.046 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2023-12-12T02:19:06.049 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2023-12-12T02:19:06.049 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2023-12-12T02:19:06.622 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~108s 2023-12-12T02:19:07.532 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:19:07.548 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:19:07.549 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:07.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-12T02:19:07.597 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:19:07.598 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_9 deleted 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:19:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_9 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_9 2023-12-12T02:19:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:07.746 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_9: (2) No such file or directory 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_9 2023-12-12T02:19:07.750 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2023-12-12T02:19:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_9 2023-12-12T02:19:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.801 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:19:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:07.802 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:08.122 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:08.123 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:19:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:09.127 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:09.128 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:09.233 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:09.236 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:11.238 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:11.294 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:11.343 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:11.347 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:19:12.026 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~113s 2023-12-12T02:19:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:15.348 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:15.405 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:15.455 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:15.458 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:19:17.431 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~119s 2023-12-12T02:19:22.835 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~124s 2023-12-12T02:19:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:23.460 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:23.571 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:19:28.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~130s 2023-12-12T02:19:33.643 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~135s 2023-12-12T02:19:39.047 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~141s 2023-12-12T02:19:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_9 2023-12-12T02:19:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_9 2023-12-12T02:19:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2023-12-12T02:19:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2023-12-12T02:19:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image=image_10 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ create_image cluster2 mirror image_10 512M 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local size=128 2023-12-12T02:19:39.860 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -n 512M ] 2023-12-12T02:19:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ size=512M 2023-12-12T02:19:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ shift 2023-12-12T02:19:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering mirror/image_10 2023-12-12T02:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ enable_mirror cluster2 mirror image_10 2023-12-12T02:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:39.908 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local mode=snapshot 2023-12-12T02:19:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2023-12-12T02:19:40.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring enabled 2023-12-12T02:19:40.536 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2023-12-12T02:19:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:rbd image 'image_10': 2023-12-12T02:19:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: size 512 MiB in 128 objects 2023-12-12T02:19:40.582 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: order 22 (4 MiB objects) 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: snapshot_count: 1 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: id: 1a55bd3f4f10 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: block_name_prefix: rbd_data.1a55bd3f4f10 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: format: 2 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: features: layering 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: op_features: 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: flags: 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: create_timestamp: Tue Dec 12 02:19:39 2023 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: access_timestamp: Tue Dec 12 02:19:39 2023 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: modify_timestamp: Tue Dec 12 02:19:39 2023 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring state: enabled 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring mode: snapshot 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring global id: 2c29c26e-b19c-4ee1-b7ad-593c2220b2d3 2023-12-12T02:19:40.583 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout: mirroring primary: true 2023-12-12T02:19:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stress_write_image cluster2 mirror image_10 2023-12-12T02:19:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:40.586 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:40.587 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ awk BEGIN {srand(); print int(10 * rand()) + 5} 2023-12-12T02:19:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local duration=10 2023-12-12T02:19:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:19:40.588 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2023-12-12T02:19:40.781 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:100 IOs, 51584000 bytes 2023-12-12T02:19:40.913 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:200 IOs, 89913344 bytes 2023-12-12T02:19:41.068 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:300 IOs, 147962880 bytes 2023-12-12T02:19:41.291 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:400 IOs, 254856192 bytes 2023-12-12T02:19:41.396 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:500 IOs, 306920448 bytes 2023-12-12T02:19:41.592 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:600 IOs, 369560576 bytes 2023-12-12T02:19:41.702 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:700 IOs, 401594368 bytes 2023-12-12T02:19:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:800 IOs, 431577088 bytes 2023-12-12T02:19:42.021 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:900 IOs, 512727040 bytes 2023-12-12T02:19:42.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1000 IOs, 536956928 bytes 2023-12-12T02:19:42.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1100 IOs, 564628480 bytes 2023-12-12T02:19:42.356 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1200 IOs, 585696256 bytes 2023-12-12T02:19:42.448 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1300 IOs, 586295296 bytes 2023-12-12T02:19:42.538 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1400 IOs, 596465664 bytes 2023-12-12T02:19:42.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1500 IOs, 621560832 bytes 2023-12-12T02:19:42.744 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1600 IOs, 630558720 bytes 2023-12-12T02:19:42.863 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1700 IOs, 672809984 bytes 2023-12-12T02:19:42.965 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1800 IOs, 691175424 bytes 2023-12-12T02:19:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:1900 IOs, 722887680 bytes 2023-12-12T02:19:43.246 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2000 IOs, 797948928 bytes 2023-12-12T02:19:43.373 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2100 IOs, 832433152 bytes 2023-12-12T02:19:43.497 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2200 IOs, 866560000 bytes 2023-12-12T02:19:43.634 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2300 IOs, 929026048 bytes 2023-12-12T02:19:43.791 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2400 IOs, 969339904 bytes 2023-12-12T02:19:43.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2500 IOs, 1008335872 bytes 2023-12-12T02:19:44.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2600 IOs, 1030622208 bytes 2023-12-12T02:19:44.149 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2700 IOs, 1079400448 bytes 2023-12-12T02:19:44.284 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2800 IOs, 1107380224 bytes 2023-12-12T02:19:44.404 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:2900 IOs, 1137074176 bytes 2023-12-12T02:19:44.450 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~146s 2023-12-12T02:19:44.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3000 IOs, 1168242688 bytes 2023-12-12T02:19:44.656 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3100 IOs, 1204182016 bytes 2023-12-12T02:19:44.794 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3200 IOs, 1241997312 bytes 2023-12-12T02:19:45.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3300 IOs, 1340800000 bytes 2023-12-12T02:19:45.175 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3400 IOs, 1391638528 bytes 2023-12-12T02:19:45.312 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3500 IOs, 1440429056 bytes 2023-12-12T02:19:45.426 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3600 IOs, 1470714880 bytes 2023-12-12T02:19:45.507 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3700 IOs, 1474357248 bytes 2023-12-12T02:19:45.640 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3800 IOs, 1525710848 bytes 2023-12-12T02:19:45.780 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:3900 IOs, 1550219264 bytes 2023-12-12T02:19:45.956 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4000 IOs, 1626558464 bytes 2023-12-12T02:19:46.042 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4100 IOs, 1627199488 bytes 2023-12-12T02:19:46.219 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4200 IOs, 1731530752 bytes 2023-12-12T02:19:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4300 IOs, 1801681920 bytes 2023-12-12T02:19:46.495 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4400 IOs, 1810007040 bytes 2023-12-12T02:19:46.673 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4500 IOs, 1884894208 bytes 2023-12-12T02:19:46.823 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4600 IOs, 1955842048 bytes 2023-12-12T02:19:46.960 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4700 IOs, 2033909760 bytes 2023-12-12T02:19:47.193 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4800 IOs, 2097639424 bytes 2023-12-12T02:19:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:4900 IOs, 2134208512 bytes 2023-12-12T02:19:47.446 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5000 IOs, 2163194880 bytes 2023-12-12T02:19:47.589 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5100 IOs, 2190607360 bytes 2023-12-12T02:19:47.763 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5200 IOs, 2261586944 bytes 2023-12-12T02:19:47.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5300 IOs, 2300201984 bytes 2023-12-12T02:19:48.036 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5400 IOs, 2340136960 bytes 2023-12-12T02:19:48.215 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5500 IOs, 2406994944 bytes 2023-12-12T02:19:48.349 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5600 IOs, 2445647872 bytes 2023-12-12T02:19:48.506 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5700 IOs, 2497828864 bytes 2023-12-12T02:19:48.698 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5800 IOs, 2590164992 bytes 2023-12-12T02:19:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:5900 IOs, 2661031936 bytes 2023-12-12T02:19:49.028 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6000 IOs, 2726574080 bytes 2023-12-12T02:19:49.137 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6100 IOs, 2747237376 bytes 2023-12-12T02:19:49.322 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6200 IOs, 2836539392 bytes 2023-12-12T02:19:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6300 IOs, 2874220544 bytes 2023-12-12T02:19:49.576 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6400 IOs, 2915930112 bytes 2023-12-12T02:19:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6500 IOs, 2941399040 bytes 2023-12-12T02:19:49.854 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~151s 2023-12-12T02:19:49.901 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6600 IOs, 3015751680 bytes 2023-12-12T02:19:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6700 IOs, 3046678528 bytes 2023-12-12T02:19:50.171 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6800 IOs, 3094912000 bytes 2023-12-12T02:19:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:6900 IOs, 3137035264 bytes 2023-12-12T02:19:50.410 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7000 IOs, 3166823424 bytes 2023-12-12T02:19:50.543 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:7100 IOs, 3197012992 bytes 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=124 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 124 -eq 124 ] 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=present 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:19:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id=162bb42e2583 2023-12-12T02:19:50.650 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_10 present 162bb42e2583 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=present 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id=162bb42e2583 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:19:50.751 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:19:50.752 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id=162bb42e2583 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 162bb42e2583 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -z 162bb42e2583 -o 162bb42e2583 = 162bb42e2583 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_state=present 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test present = present 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ disable_mirror cluster2 mirror image_10 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:50.813 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:50.814 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2023-12-12T02:19:51.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:19:51.611+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646789000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=386 2023-12-12T02:19:51.611 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:19:51.611+0000 7f41396e7640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55e646789000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=386, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=0d58dc8c-c526-4d8f-b076-f69b67a71374, primary_snap_id=182, last_copied_object_number=128, snap_seqs={386=18446744073709551614}] 2023-12-12T02:19:52.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:19:52.123+0000 7fde2e7ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55a1d151d520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2023-12-12T02:19:52.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:19:52.123+0000 7fde34ff9640 -1 rbd::mirror::ImageReplayer: 0x55a1d0777900 [3/2c29c26e-b19c-4ee1-b7ad-593c2220b2d3] operator(): start failed: (85) Interrupted system call should be restarted 2023-12-12T02:19:52.139 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9326727004575885) > weight (0.25) 2023-12-12T02:19:52.139 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2023-12-12T02:19:52.139 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2023-12-12T02:19:52.139 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2023-12-12T02:19:52.140 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:19:52.139+0000 7f49a7dcb640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 67908) UID: 1000 2023-12-12T02:19:55.258 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~157s 2023-12-12T02:19:56.616 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:Mirroring disabled 2023-12-12T02:19:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2023-12-12T02:19:56.632 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local state=deleted 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local s 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:19:56.633 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:56.678 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-12T02:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ image_id= 2023-12-12T02:19:56.683 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0.1 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test_image_present cluster1 mirror image_10 deleted 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local test_state=deleted 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image_id= 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_state=deleted 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local current_image_id 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ get_image_id cluster1 mirror image_10 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:19:56.785 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster1 info mirror/image_10 2023-12-12T02:19:56.786 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sed -ne s/^.*block_name_prefix: rbd_data\.//p 2023-12-12T02:19:56.841 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error opening image image_10: (2) No such file or directory 2023-12-12T02:19:56.844 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ current_image_id= 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test deleted = deleted 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ purge_snapshots cluster2 mirror image_10 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.845 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2023-12-12T02:19:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image_retry cluster2 mirror image_10 2023-12-12T02:19:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.894 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:19:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:19:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:19:56.941 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:19:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:56.986 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:56.989 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 1 2023-12-12T02:19:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:19:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:19:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:19:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:19:57.991 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:19:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:19:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:19:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:19:58.098 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:19:58.101 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 2 2023-12-12T02:19:58.141 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2023-12-12T02:19:58.141 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:19:58.141 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2023-12-12T02:19:58.141 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2023-12-12T02:19:58.141 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2023-12-12T02:19:58.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2023-12-12T02:19:58.144 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2023-12-12T02:20:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:20:00.102 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:20:00.103 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:20:00.159 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:20:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:20:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:20:00.207 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:20:00.210 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 4 2023-12-12T02:20:00.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~162s 2023-12-12T02:20:04.211 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:20:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:20:04.212 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:20:04.266 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:20:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:20:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:20:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:20:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 8 2023-12-12T02:20:06.066 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~168s 2023-12-12T02:20:11.470 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~173s 2023-12-12T02:20:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:20:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:20:12.319 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:20:12.376 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:20:12.424 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 0% complete...failed. 2023-12-12T02:20:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:rbd: error: image still has watchers 2023-12-12T02:20:12.425 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout. 2023-12-12T02:20:12.428 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 16 2023-12-12T02:20:16.874 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~178s 2023-12-12T02:20:19.050 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.39792490626758337) > weight (0.25) 2023-12-12T02:20:19.050 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2023-12-12T02:20:19.050 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2023-12-12T02:20:19.051 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2023-12-12T02:20:19.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:19.050+0000 7f4146f02640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 67714) UID: 1000 2023-12-12T02:20:21.051 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2023-12-12T02:20:21.051 INFO:teuthology.orchestra.run:waiting for 600 2023-12-12T02:20:21.052 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2023-12-12T02:20:21.052 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2023-12-12T02:20:21.052 DEBUG:teuthology.orchestra.run.smithi140:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2023-12-12T02:20:21.055 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2023-12-12T02:20:21.055 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2023-12-12T02:20:22.278 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~184s 2023-12-12T02:20:27.681 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~189s 2023-12-12T02:20:28.430 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ remove_image cluster2 mirror image_10 2023-12-12T02:20:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local image=image_10 2023-12-12T02:20:28.431 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2023-12-12T02:20:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2023-12-12T02:20:29.012 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2023-12-12T02:20:29.018 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ testlog TEST: check if removed images' OMAP are removed 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ tee -a /tmp/tmp.6PQoIp3L40/rbd-mirror.test.log 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ date +%F %T 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo 2023-12-12 02:20:29 TEST: check if removed images' OMAP are removed 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:2023-12-12 02:20:29 TEST: check if removed images' OMAP are removed 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_in_omap cluster1 mirror 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.019 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=status_global 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=status_global 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-12T02:20:29.020 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c status_global 2023-12-12T02:20:29.062 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.065 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_ 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_ 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster1 mirror rbd_mirroring 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2023-12-12T02:20:29.066 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c image_ 2023-12-12T02:20:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.104 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_map 2023-12-12T02:20:29.105 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2023-12-12T02:20:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.106 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_map 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster1 mirror rbd_mirror_leader 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2023-12-12T02:20:29.107 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c image_map 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_image_in_omap cluster2 mirror 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=status_global 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=status_global 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-12T02:20:29.147 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c status_global 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_ 2023-12-12T02:20:29.180 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_ 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster2 mirror rbd_mirroring 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirroring 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2023-12-12T02:20:29.182 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c image_ 2023-12-12T02:20:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.216 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_map 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ sleep 0 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local filter=image_map 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ list_omap_keys cluster2 mirror rbd_mirror_leader 2023-12-12T02:20:29.217 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local pool=mirror 2023-12-12T02:20:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local obj_name=rbd_mirror_leader 2023-12-12T02:20:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2023-12-12T02:20:29.218 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ grep -c image_map 2023-12-12T02:20:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test 0 = 0 2023-12-12T02:20:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ error_code=0 2023-12-12T02:20:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set -e 2023-12-12T02:20:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 0 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ cleanup 0 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local error_code=0 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ set +e 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -ne 0 ] 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z ] 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster instance 2023-12-12T02:20:29.251 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-12T02:20:30.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.490+0000 7fd8085c7640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5608609c4480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:30.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.490+0000 7fd8085c7640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5608607a4900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:30.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.490+0000 7fd808dc8640 -1 rbd::mirror::PoolWatcher: 0x56085f6143c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:30.489 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.490+0000 7fd808dc8640 -1 rbd::mirror::PoolWatcher: 0x56085f6152c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:30.495 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.498+0000 7fd80edd4640 -1 rbd::mirror::InstanceWatcher: 0x5608607d9500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6094 object: (2) No such file or directory 2023-12-12T02:20:30.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.498+0000 7fd80f5d5640 -1 rbd::mirror::InstanceWatcher: 0x560860a0dc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6094 object: (2) No such file or directory 2023-12-12T02:20:30.496 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.498+0000 7fd8085c7640 -1 rbd::mirror::InstanceWatcher: 0x5608607d9500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:30.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.506+0000 7fd808dc8640 -1 rbd::mirror::InstanceWatcher: 0x560860a0dc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:30.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.506+0000 7fd808dc8640 -1 rbd::mirror::MirrorStatusUpdater 0x560860844000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:30.504 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.506+0000 7fd80edd4640 -1 rbd::mirror::NamespaceReplayer: 0x5608607d6820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:30.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.510+0000 7fd8085c7640 -1 rbd::mirror::MirrorStatusUpdater 0x560860844120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:30.510 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.510+0000 7fd80f5d5640 -1 rbd::mirror::NamespaceReplayer: 0x5608607d69c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror' does not exist 2023-12-12T02:20:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2023-12-12T02:20:30.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.650+0000 7fd808dc8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560860b7c480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:30.648 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.650+0000 7fd8085c7640 -1 rbd::mirror::PoolWatcher: 0x56085f6150e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:30.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.654+0000 7fd80f5d5640 -1 rbd::mirror::LeaderWatcher: 0x5608605a5b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-12T02:20:30.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.654+0000 7fd80f5d5640 -1 rbd::mirror::LeaderWatcher: 0x5608605a5b00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-12T02:20:30.652 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.654+0000 7fd80edd4640 -1 rbd::mirror::LeaderWatcher: 0x5608605a5b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:30.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.654+0000 7fd80edd4640 -1 rbd::mirror::InstanceWatcher: 0x56085f669c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6094 object: (2) No such file or directory 2023-12-12T02:20:30.653 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.654+0000 7fd8085c7640 -1 rbd::mirror::InstanceWatcher: 0x56085f669c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:30.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.658+0000 7fd808dc8640 -1 rbd::mirror::MirrorStatusUpdater 0x5608611a4b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:30.656 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:30.658+0000 7fd80edd4640 -1 rbd::mirror::NamespaceReplayer: 0x5608607d6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:31.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.338+0000 7f56257ef640 -1 rbd::mirror::InstanceWatcher: 0x5618bd048a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6004 object: (2) No such file or directory 2023-12-12T02:20:31.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.338+0000 7f56257ef640 -1 rbd::mirror::InstanceWatcher: 0x5618bcfc9500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6004 object: (2) No such file or directory 2023-12-12T02:20:31.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.342+0000 7f561efe2640 -1 rbd::mirror::InstanceWatcher: 0x5618bd048a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:31.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.342+0000 7f561f7e3640 -1 rbd::mirror::InstanceWatcher: 0x5618bcfc9500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:31.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.346+0000 7f561efe2640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bd028120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:31.345 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.346+0000 7f5625ff0640 -1 rbd::mirror::NamespaceReplayer: 0x5618bcfc69c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:31.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.350+0000 7f561f7e3640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bd028000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:31.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.350+0000 7f56257ef640 -1 rbd::mirror::NamespaceReplayer: 0x5618bcfc6820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:31.563 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.566+0000 7f56257ef640 -1 rbd::mirror::LeaderWatcher: 0x5618bda39b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:31.564 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.566+0000 7f56257ef640 -1 rbd::mirror::InstanceWatcher: 0x5618bcc41880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6004 object: (2) No such file or directory 2023-12-12T02:20:31.566 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.566+0000 7f561efe2640 -1 rbd::mirror::InstanceWatcher: 0x5618bcc41880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:31.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.570+0000 7f56187d5640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bcd9c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:31.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.570+0000 7f5625ff0640 -1 rbd::mirror::NamespaceReplayer: 0x5618bcfc6000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:31.567 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.570+0000 7f561f7e3640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bd9d4c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:31.568 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:31.570+0000 7f5625ff0640 -1 rbd::mirror::NamespaceReplayer: 0x5618bcfc6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:31.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror' does not exist 2023-12-12T02:20:31.777 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-12T02:20:33.084 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~195s 2023-12-12T02:20:33.204 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde347f8640 -1 rbd::mirror::LeaderWatcher: 0x55a1d15bdb00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55a1d1280b40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55a1d1280000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55a1d1280900 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f7a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f7860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde2dfeb640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f70e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde27fdf640 -1 rbd::mirror::PoolWatcher: 0x55a1d1318000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde27fdf640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f7c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.205 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde277de640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f7e00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:33.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde347f8640 -1 rbd::mirror::LeaderWatcher: 0x55a1d15bdb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-12T02:20:33.206 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:33.206+0000 7fde347f8640 -1 rbd::mirror::LeaderWatcher: 0x55a1d15bdb00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2023-12-12T02:20:33.525 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror_parent' does not exist 2023-12-12T02:20:33.541 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ CEPH_ARGS= ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2023-12-12T02:20:34.699 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:pool 'mirror_parent' does not exist 2023-12-12T02:20:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirrors cluster1 2023-12-12T02:20:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1 2023-12-12T02:20:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:20:34.717 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster1:0 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:0 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster1:1 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:1 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster1:2 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:2 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster1:3 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster1:3 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirrors cluster2 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ seq 0 3 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster2:0 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:0 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster2:1 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:1 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster2:2 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:2 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ stop_mirror cluster2:3 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local cluster=cluster2:3 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ local sig= 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test -n 1 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ return 2023-12-12T02:20:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ -z 1 ] 2023-12-12T02:20:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ test = /tmp/tmp.6PQoIp3L40 2023-12-12T02:20:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ rm -Rf /tmp/tmp.6PQoIp3L40 2023-12-12T02:20:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ [ 0 -eq 0 ] 2023-12-12T02:20:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ echo OK 2023-12-12T02:20:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stdout:OK 2023-12-12T02:20:34.722 INFO:tasks.workunit.cluster1.client.mirror.smithi140.stderr:+ exit 0 2023-12-12T02:20:34.723 INFO:teuthology.orchestra.run:Running command with timeout 3600 2023-12-12T02:20:34.723 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2023-12-12T02:20:34.775 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2023-12-12T02:20:34.776 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2023-12-12T02:20:35.102 DEBUG:teuthology.parallel:result is None 2023-12-12T02:20:35.102 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-12T02:20:35.111 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-12T02:20:35.111 DEBUG:teuthology.orchestra.run.smithi140:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2023-12-12T02:20:35.167 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2023-12-12T02:20:35.167 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-12T02:20:35.179 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-12T02:20:35.179 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2023-12-12T02:20:35.179 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-12T02:20:35.179 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2023-12-12T02:20:35.189 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2023-12-12T02:20:35.189 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 19 2023-12-12T02:20:35.189 INFO:tasks.rbd_mirror_thrash:done joining 2023-12-12T02:20:35.189 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.199 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2023-12-12T02:20:35.200 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde357fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 45984) UID: 1000 2023-12-12T02:20:35.211 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde20fd1640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55a1d174f680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:35.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde207d0640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f63c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:35.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde347f8640 -1 rbd::mirror::InstanceWatcher: 0x55a1d097d500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4555 object: (2) No such file or directory 2023-12-12T02:20:35.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde347f8640 -1 rbd::mirror::InstanceWatcher: 0x55a1d09e4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4555 object: (2) No such file or directory 2023-12-12T02:20:35.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde1a7c4640 -1 rbd::mirror::PoolWatcher: 0x55a1cf9f61e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde2dfeb640 -1 rbd::mirror::InstanceWatcher: 0x55a1d097d500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde2dfeb640 -1 rbd::mirror::InstanceWatcher: 0x55a1d09e4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde34ff9640 -1 rbd::mirror::LeaderWatcher: 0x55a1d0d78300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde34ff9640 -1 rbd::mirror::LeaderWatcher: 0x55a1d0d78300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde34ff9640 -1 rbd::mirror::LeaderWatcher: 0x55a1d0d78300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde27fdf640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d09b85a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde277de640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d09b8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde34ff9640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0958820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde34ff9640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d09589c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde2dfeb640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d09b8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde347f8640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0958820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde347f8640 -1 rbd::mirror::InstanceWatcher: 0x55a1d097ce00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4558 object: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde2e7ec640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d09b8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.214+0000 7fde347f8640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d09589c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.218+0000 7fde20fd1640 -1 rbd::mirror::InstanceWatcher: 0x55a1d097ce00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.218+0000 7fde19fc3640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d0c065a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.218+0000 7fde347f8640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0d7c4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.218+0000 7fde207d0640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d0beaa20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.215 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.218+0000 7fde34ff9640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0d7c4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.218 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde347f8640 -1 rbd::mirror::LeaderWatcher: 0x55a1d15bdb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde347f8640 -1 rbd::mirror::InstanceWatcher: 0x55a1d08f3880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4555 object: (2) No such file or directory 2023-12-12T02:20:35.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde2dfeb640 -1 rbd::mirror::InstanceWatcher: 0x55a1d08f3880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde27fdf640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d1766480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde34ff9640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0958000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde2e7ec640 -1 rbd::mirror::MirrorStatusUpdater 0x55a1d15bed80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.222+0000 7fde34ff9640 -1 rbd::mirror::NamespaceReplayer: 0x55a1d0958000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2023-12-12T02:20:35.238 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.247 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2023-12-12T02:20:35.248 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5963f3f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 61577) UID: 1000 2023-12-12T02:20:35.280 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5962f3d640 -1 rbd::mirror::LeaderWatcher: 0x56128f6ea300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5962f3d640 -1 rbd::mirror::InstanceWatcher: 0x56128f2e7500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5654 object: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5962f3d640 -1 rbd::mirror::InstanceWatcher: 0x56128f365500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5654 object: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5962f3d640 -1 rbd::mirror::InstanceWatcher: 0x56128f364e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5657 object: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f595c730640 -1 rbd::mirror::InstanceWatcher: 0x56128f2e7500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f594f716640 -1 rbd::mirror::InstanceWatcher: 0x56128f364e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.281 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f595c730640 -1 rbd::mirror::InstanceWatcher: 0x56128f365500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5955f23640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f3265a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.282+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5956724640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f326360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5948f09640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f5805a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f640680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f595cf31640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f326000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f594ef15640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f562900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f640680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f595c730640 -1 rbd::mirror::MirrorStatusUpdater 0x56128f326120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.283 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f596373e640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.286+0000 7f596373e640 -1 rbd::mirror::LeaderWatcher: 0x56128ff35b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f596373e640 -1 rbd::mirror::InstanceWatcher: 0x56128f291880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5654 object: (2) No such file or directory 2023-12-12T02:20:35.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f595cf31640 -1 rbd::mirror::InstanceWatcher: 0x56128f291880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f5955f23640 -1 rbd::mirror::MirrorStatusUpdater 0x5612900ea480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f595c730640 -1 rbd::mirror::MirrorStatusUpdater 0x56128ff3ec60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.288 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.290+0000 7f5962f3d640 -1 rbd::mirror::NamespaceReplayer: 0x56128f2ce000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.304 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2023-12-12T02:20:35.305 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.315 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2023-12-12T02:20:35.315 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a79f0b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 77641) UID: 1000 2023-12-12T02:20:35.347 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::LeaderWatcher: 0x563d1ffea300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a78f09640 -1 rbd::mirror::InstanceWatcher: 0x563d1fbd2700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6775 object: (2) No such file or directory 2023-12-12T02:20:35.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a78f09640 -1 rbd::mirror::InstanceWatcher: 0x563d1fcdb180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6775 object: (2) No such file or directory 2023-12-12T02:20:35.348 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::InstanceWatcher: 0x563d1fc62e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6778 object: (2) No such file or directory 2023-12-12T02:20:35.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a726fc640 -1 rbd::mirror::InstanceWatcher: 0x563d1fbd2700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a726fc640 -1 rbd::mirror::InstanceWatcher: 0x563d1fcdb180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a64ee1640 -1 rbd::mirror::InstanceWatcher: 0x563d1fc62e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.349 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a6c6f0640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fc245a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a78f09640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd0820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a6beef640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fc24360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd09c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a5eed5640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fc246c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::NamespaceReplayer: 0x563d1ffee4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a72efd640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fc24000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a78f09640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd0820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a726fc640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fc24120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd09c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a656e2640 -1 rbd::mirror::MirrorStatusUpdater 0x563d1fe4a900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.350 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.350+0000 7f5a7970a640 -1 rbd::mirror::NamespaceReplayer: 0x563d1ffee4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a7970a640 -1 rbd::mirror::LeaderWatcher: 0x563d2082fb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.352 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a78f09640 -1 rbd::mirror::InstanceWatcher: 0x563d1fb6d880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6775 object: (2) No such file or directory 2023-12-12T02:20:35.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a72efd640 -1 rbd::mirror::InstanceWatcher: 0x563d1fb6d880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a6c6f0640 -1 rbd::mirror::MirrorStatusUpdater 0x563d209ec480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a7970a640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a726fc640 -1 rbd::mirror::MirrorStatusUpdater 0x563d2083ec60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.354+0000 7f5a78f09640 -1 rbd::mirror::NamespaceReplayer: 0x563d1fbd0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.368 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2023-12-12T02:20:35.368 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.378 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2023-12-12T02:20:35.378 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff843039640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 75821) UID: 1000 2023-12-12T02:20:35.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842037640 -1 rbd::mirror::LeaderWatcher: 0x555edea98300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842838640 -1 rbd::mirror::InstanceWatcher: 0x555ede72a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6606 object: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842838640 -1 rbd::mirror::InstanceWatcher: 0x555ede72b880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6606 object: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842037640 -1 rbd::mirror::InstanceWatcher: 0x555ede72a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6608 object: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83b82a640 -1 rbd::mirror::InstanceWatcher: 0x555ede72a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83b82a640 -1 rbd::mirror::InstanceWatcher: 0x555ede72b880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.412 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff82e00f640 -1 rbd::mirror::InstanceWatcher: 0x555ede72a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83501d640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede796360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842037640 -1 rbd::mirror::NamespaceReplayer: 0x555ede724820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83581e640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede7965a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842838640 -1 rbd::mirror::NamespaceReplayer: 0x555ede7249c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff827802640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede7966c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842037640 -1 rbd::mirror::NamespaceReplayer: 0x555edea76820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83c02b640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede796000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842838640 -1 rbd::mirror::NamespaceReplayer: 0x555ede724820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff83b82a640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede796120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842838640 -1 rbd::mirror::NamespaceReplayer: 0x555ede7249c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff82e810640 -1 rbd::mirror::MirrorStatusUpdater 0x555ede994900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.414 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.414+0000 7ff842037640 -1 rbd::mirror::NamespaceReplayer: 0x555edea76820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.415 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff842037640 -1 rbd::mirror::LeaderWatcher: 0x555edf375b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff842037640 -1 rbd::mirror::InstanceWatcher: 0x555ede6ad880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6606 object: (2) No such file or directory 2023-12-12T02:20:35.416 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff83c02b640 -1 rbd::mirror::InstanceWatcher: 0x555ede6ad880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff83501d640 -1 rbd::mirror::MirrorStatusUpdater 0x555edf35c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff842037640 -1 rbd::mirror::NamespaceReplayer: 0x555ede724000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff83b82a640 -1 rbd::mirror::MirrorStatusUpdater 0x555edf2f4a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.418+0000 7ff842838640 -1 rbd::mirror::NamespaceReplayer: 0x555ede724000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.431 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2023-12-12T02:20:35.431 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.442 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2023-12-12T02:20:35.442 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03b488640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 76894) UID: 1000 2023-12-12T02:20:35.475 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03a486640 -1 rbd::mirror::LeaderWatcher: 0x558d0d89c300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::InstanceWatcher: 0x558d0d509c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6167 object: (2) No such file or directory 2023-12-12T02:20:35.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::InstanceWatcher: 0x558d0d502a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6167 object: (2) No such file or directory 2023-12-12T02:20:35.476 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::InstanceWatcher: 0x558d0d503180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6170 object: (2) No such file or directory 2023-12-12T02:20:35.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03447a640 -1 rbd::mirror::InstanceWatcher: 0x558d0d509c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.477 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03447a640 -1 rbd::mirror::InstanceWatcher: 0x558d0d502a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe02d46c640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d56c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe02645e640 -1 rbd::mirror::InstanceWatcher: 0x558d0d503180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe02d46c640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d56c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe033c79640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d56c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe033c79640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d56c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe01fc51640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d56c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d888680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.478 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe026c5f640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0d766c60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.479 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.478+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d888680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.480 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe03ac87640 -1 rbd::mirror::LeaderWatcher: 0x558d0e151b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe03ac87640 -1 rbd::mirror::InstanceWatcher: 0x558d0d381880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6167 object: (2) No such file or directory 2023-12-12T02:20:35.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe03447a640 -1 rbd::mirror::InstanceWatcher: 0x558d0d381880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe02dc6d640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0e310480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe03ac87640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe033c79640 -1 rbd::mirror::MirrorStatusUpdater 0x558d0e154d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi140.stderr:2023-12-12T02:20:35.482+0000 7fe03a486640 -1 rbd::mirror::NamespaceReplayer: 0x558d0d4fa000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.497 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2023-12-12T02:20:35.497 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.508 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2023-12-12T02:20:35.508 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd80fdd6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 75328) UID: 1000 2023-12-12T02:20:35.539 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd80f5d5640 -1 rbd::mirror::LeaderWatcher: 0x560860b86300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd80f5d5640 -1 rbd::mirror::InstanceWatcher: 0x560860cd4000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6096 object: (2) No such file or directory 2023-12-12T02:20:35.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd7fadac640 -1 rbd::mirror::InstanceWatcher: 0x560860cd4000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd7f4da0640 -1 rbd::mirror::MirrorStatusUpdater 0x560860a627e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.541 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd80f5d5640 -1 rbd::mirror::NamespaceReplayer: 0x560860b884e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd7fb5ad640 -1 rbd::mirror::MirrorStatusUpdater 0x5608609f47e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi140.stderr:2023-12-12T02:20:35.542+0000 7fd80f5d5640 -1 rbd::mirror::NamespaceReplayer: 0x560860b884e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.571 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2023-12-12T02:20:35.571 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.581 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2023-12-12T02:20:35.581 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.614 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e093b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 77857) UID: 1000 2023-12-12T02:20:35.615 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77df939640 -1 rbd::mirror::LeaderWatcher: 0x559e21e5a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77df939640 -1 rbd::mirror::InstanceWatcher: 0x559e21a72e00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6201 object: (2) No such file or directory 2023-12-12T02:20:35.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e013a640 -1 rbd::mirror::InstanceWatcher: 0x559e21a50000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6201 object: (2) No such file or directory 2023-12-12T02:20:35.616 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77df939640 -1 rbd::mirror::InstanceWatcher: 0x559e21aa1500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6205 object: (2) No such file or directory 2023-12-12T02:20:35.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d912c640 -1 rbd::mirror::InstanceWatcher: 0x559e21a72e00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.617 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d992d640 -1 rbd::mirror::InstanceWatcher: 0x559e21a50000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77cb911640 -1 rbd::mirror::InstanceWatcher: 0x559e21aa1500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d291f640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21abe360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d291f640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21abe5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e013a640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a820 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77c5104640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21ea0480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e013a640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a9c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e013a640 -1 rbd::mirror::NamespaceReplayer: 0x559e21e5e4e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d912c640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21abe000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77e013a640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a820 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77d992d640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21abe120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.618 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77cc112640 -1 rbd::mirror::MirrorStatusUpdater 0x559e21cc4a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77df939640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a9c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.619 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.618+0000 7f77df939640 -1 rbd::mirror::NamespaceReplayer: 0x559e21e5e4e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.620 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77e013a640 -1 rbd::mirror::LeaderWatcher: 0x559e226c1b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77df939640 -1 rbd::mirror::InstanceWatcher: 0x559e219c7880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6201 object: (2) No such file or directory 2023-12-12T02:20:35.621 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77d912c640 -1 rbd::mirror::InstanceWatcher: 0x559e219c7880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77d3120640 -1 rbd::mirror::MirrorStatusUpdater 0x559e226a66c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77df939640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77d992d640 -1 rbd::mirror::MirrorStatusUpdater 0x559e2262ca20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.622 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi140.stderr:2023-12-12T02:20:35.622+0000 7f77e013a640 -1 rbd::mirror::NamespaceReplayer: 0x559e21a4a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2023-12-12T02:20:35.635 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2023-12-12T02:20:35.647 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2023-12-12T02:20:35.647 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:20:35.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f56267f1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 72702) UID: 1000 2023-12-12T02:20:35.679 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f5611fc8640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5618bcf0cb40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2023-12-12T02:20:35.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f560afba640 -1 rbd::mirror::PoolWatcher: 0x5618bbe521e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:35.680 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f56117c7640 -1 rbd::mirror::PoolWatcher: 0x5618bbe523c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2023-12-12T02:20:35.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f5625ff0640 -1 rbd::mirror::LeaderWatcher: 0x5618bd3d0300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2023-12-12T02:20:35.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f5625ff0640 -1 rbd::mirror::LeaderWatcher: 0x5618bd3d0300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2023-12-12T02:20:35.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f56257ef640 -1 rbd::mirror::LeaderWatcher: 0x5618bd3d0300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2023-12-12T02:20:35.681 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.682+0000 7f56257ef640 -1 rbd::mirror::InstanceWatcher: 0x5618bd494380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6007 object: (2) No such file or directory 2023-12-12T02:20:35.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.686+0000 7f56117c7640 -1 rbd::mirror::InstanceWatcher: 0x5618bd494380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2023-12-12T02:20:35.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.686+0000 7f560b7bb640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bd2665a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.682 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.686+0000 7f56257ef640 -1 rbd::mirror::NamespaceReplayer: 0x5618bd3d44e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.686+0000 7f5611fc8640 -1 rbd::mirror::MirrorStatusUpdater 0x5618bd252a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2023-12-12T02:20:35.683 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi140.stderr:2023-12-12T02:20:35.686+0000 7f5625ff0640 -1 rbd::mirror::NamespaceReplayer: 0x5618bd3d44e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2023-12-12T02:20:35.701 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2023-12-12T02:20:35.701 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-12T02:20:35.712 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2023-12-12T02:20:35.712 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:36.030 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:36.030 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:36.049 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1498,"stamp":"2023-12-12T02:20:34.530638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2996748,"kb_used_data":369808,"kb_used_omap":291,"kb_used_meta":2626588,"kb_avail":280118772,"statfs":{"total":289910292480,"available":286841622528,"internally_reserved":0,"allocated":378683392,"data_stored":374178844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2689626666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536833078,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113602,"num_read_kb":-39868702,"num_write":-397940,"num_write_kb":-144697292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.093588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.592626+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:19:51.592626+0000","last_peered":"2023-12-12T02:19:51.592626+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:19:51.592626+0000","last_undegraded":"2023-12-12T02:19:51.592626+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.593769+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:19:51.593769+0000","last_peered":"2023-12-12T02:19:51.593769+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:19:51.593769+0000","last_undegraded":"2023-12-12T02:19:51.593769+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471006+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:19:40.471006+0000","last_peered":"2023-12-12T02:19:40.471006+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:19:40.471006+0000","last_undegraded":"2023-12-12T02:19:40.471006+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471106+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:19:40.471106+0000","last_peered":"2023-12-12T02:19:40.471106+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:19:40.471106+0000","last_undegraded":"2023-12-12T02:19:40.471106+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.473854+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:19:40.473854+0000","last_peered":"2023-12-12T02:19:40.473854+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:19:40.473854+0000","last_undegraded":"2023-12-12T02:19:40.473854+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.482816+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:19:40.482816+0000","last_peered":"2023-12-12T02:19:40.482816+0000","last_clean":"2023-12-12T02:19:40.482816+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:19:40.482816+0000","last_undegraded":"2023-12-12T02:19:40.482816+0000","last_fullsized":"2023-12-12T02:19:40.482816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.486548+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:19:40.486548+0000","last_peered":"2023-12-12T02:19:40.486548+0000","last_clean":"2023-12-12T02:19:40.486548+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:19:40.486548+0000","last_undegraded":"2023-12-12T02:19:40.486548+0000","last_fullsized":"2023-12-12T02:19:40.486548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.473378+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:19:40.473378+0000","last_peered":"2023-12-12T02:19:40.473378+0000","last_clean":"2023-12-12T02:19:40.473378+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:19:40.473378+0000","last_undegraded":"2023-12-12T02:19:40.473378+0000","last_fullsized":"2023-12-12T02:19:40.473378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":529,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:19:40.469829+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:19:40.469829+0000","last_peered":"2023-12-12T02:19:40.469829+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:19:40.469829+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732552,"kb_used_data":3072,"kb_used_omap":99,"kb_used_meta":729372,"kb_avail":93639288,"statfs":{"total":96636764160,"available":95886630912,"internally_reserved":0,"allocated":3145728,"data_stored":1684161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":746877088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51600000000000001,"15min":0.55000000000000004},"min":{"1min":0.17399999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.28,"5min":4.2880000000000003,"15min":6.4779999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.61799999999999999,"15min":0.65700000000000003},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.4339999999999999,"5min":4.3259999999999996,"15min":6.5190000000000001},"last":0.55400000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877576,"kb_used_data":3648,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494264,"statfs":{"total":96636764160,"available":95738126336,"internally_reserved":0,"allocated":3735552,"data_stored":2274510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:36.050 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:36.370 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:36.370 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:36.387 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1498,"stamp":"2023-12-12T02:20:34.530638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2996748,"kb_used_data":369808,"kb_used_omap":291,"kb_used_meta":2626588,"kb_avail":280118772,"statfs":{"total":289910292480,"available":286841622528,"internally_reserved":0,"allocated":378683392,"data_stored":374178844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2689626666},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536833078,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113602,"num_read_kb":-39868702,"num_write":-397940,"num_write_kb":-144697292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.093588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.592626+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:19:51.592626+0000","last_peered":"2023-12-12T02:19:51.592626+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:19:51.592626+0000","last_undegraded":"2023-12-12T02:19:51.592626+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.593769+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:19:51.593769+0000","last_peered":"2023-12-12T02:19:51.593769+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:19:51.593769+0000","last_undegraded":"2023-12-12T02:19:51.593769+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471006+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:19:40.471006+0000","last_peered":"2023-12-12T02:19:40.471006+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:19:40.471006+0000","last_undegraded":"2023-12-12T02:19:40.471006+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471106+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:19:40.471106+0000","last_peered":"2023-12-12T02:19:40.471106+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:19:40.471106+0000","last_undegraded":"2023-12-12T02:19:40.471106+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.473854+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:19:40.473854+0000","last_peered":"2023-12-12T02:19:40.473854+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:19:40.473854+0000","last_undegraded":"2023-12-12T02:19:40.473854+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.482816+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:19:40.482816+0000","last_peered":"2023-12-12T02:19:40.482816+0000","last_clean":"2023-12-12T02:19:40.482816+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:19:40.482816+0000","last_undegraded":"2023-12-12T02:19:40.482816+0000","last_fullsized":"2023-12-12T02:19:40.482816+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":396,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.486548+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:19:40.486548+0000","last_peered":"2023-12-12T02:19:40.486548+0000","last_clean":"2023-12-12T02:19:40.486548+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:19:40.486548+0000","last_undegraded":"2023-12-12T02:19:40.486548+0000","last_fullsized":"2023-12-12T02:19:40.486548+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.473378+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:19:40.473378+0000","last_peered":"2023-12-12T02:19:40.473378+0000","last_clean":"2023-12-12T02:19:40.473378+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:19:40.473378+0000","last_undegraded":"2023-12-12T02:19:40.473378+0000","last_fullsized":"2023-12-12T02:19:40.473378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":529,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:19:40.469829+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:19:40.469829+0000","last_peered":"2023-12-12T02:19:40.469829+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:19:40.469829+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738820,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":732552,"kb_used_data":3072,"kb_used_omap":99,"kb_used_meta":729372,"kb_avail":93639288,"statfs":{"total":96636764160,"available":95886630912,"internally_reserved":0,"allocated":3145728,"data_stored":1684161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":746877088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:19:35 2023","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.51600000000000001,"15min":0.55000000000000004},"min":{"1min":0.17399999999999999,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.28,"5min":4.2880000000000003,"15min":6.4779999999999998},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.61799999999999999,"15min":0.65700000000000003},"min":{"1min":0.23999999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.4339999999999999,"5min":4.3259999999999996,"15min":6.5190000000000001},"last":0.55400000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877576,"kb_used_data":3648,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494264,"statfs":{"total":96636764160,"available":95738126336,"internally_reserved":0,"allocated":3735552,"data_stored":2274510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:36.388 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:36.656 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:36.656 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:36.672 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1499,"stamp":"2023-12-12T02:20:36.443738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":238,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3014804,"kb_used_data":369816,"kb_used_omap":291,"kb_used_meta":2644636,"kb_avail":280100716,"statfs":{"total":289910292480,"available":286823133184,"internally_reserved":0,"allocated":378691584,"data_stored":374183005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2708107818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":217,"apply_latency_ms":217,"commit_latency_ns":217000000,"apply_latency_ns":217000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536833078,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113602,"num_read_kb":-39868702,"num_write":-397940,"num_write_kb":-144697292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005075"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.592626+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:19:51.592626+0000","last_peered":"2023-12-12T02:19:51.592626+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:19:51.592626+0000","last_undegraded":"2023-12-12T02:19:51.592626+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.593769+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:19:51.593769+0000","last_peered":"2023-12-12T02:19:51.593769+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:19:51.593769+0000","last_undegraded":"2023-12-12T02:19:51.593769+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471006+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:19:40.471006+0000","last_peered":"2023-12-12T02:19:40.471006+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:19:40.471006+0000","last_undegraded":"2023-12-12T02:19:40.471006+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471106+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:19:40.471106+0000","last_peered":"2023-12-12T02:19:40.471106+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:19:40.471106+0000","last_undegraded":"2023-12-12T02:19:40.471106+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.473854+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:19:40.473854+0000","last_peered":"2023-12-12T02:19:40.473854+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:19:40.473854+0000","last_undegraded":"2023-12-12T02:19:40.473854+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.473378+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:19:40.473378+0000","last_peered":"2023-12-12T02:19:40.473378+0000","last_clean":"2023-12-12T02:19:40.473378+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:19:40.473378+0000","last_undegraded":"2023-12-12T02:19:40.473378+0000","last_fullsized":"2023-12-12T02:19:40.473378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":529,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:19:40.469829+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:19:40.469829+0000","last_peered":"2023-12-12T02:19:40.469829+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:19:40.469829+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750608,"kb_used_data":3080,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621232,"statfs":{"total":96636764160,"available":95868141568,"internally_reserved":0,"allocated":3153920,"data_stored":1688322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":204,"apply_latency_ms":204,"commit_latency_ns":204000000,"apply_latency_ns":204000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877576,"kb_used_data":3648,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494264,"statfs":{"total":96636764160,"available":95738126336,"internally_reserved":0,"allocated":3735552,"data_stored":2274510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:36.673 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 382, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:19:51.592626+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:19:51.592626+0000', 'last_peered': '2023-12-12T02:19:51.592626+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:19:51.592626+0000', 'last_undegraded': '2023-12-12T02:19:51.592626+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 382, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:19:51.593769+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:19:51.593769+0000', 'last_peered': '2023-12-12T02:19:51.593769+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:19:51.593769+0000', 'last_undegraded': '2023-12-12T02:19:51.593769+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 381, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:19:40.471006+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:19:40.471006+0000', 'last_peered': '2023-12-12T02:19:40.471006+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:19:40.471006+0000', 'last_undegraded': '2023-12-12T02:19:40.471006+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:36.675 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 399, 'reported_epoch': 381, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:19:40.471106+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:19:40.471106+0000', 'last_peered': '2023-12-12T02:19:40.471106+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:19:40.471106+0000', 'last_undegraded': '2023-12-12T02:19:40.471106+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.675 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:36.675 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 16, 'reported_epoch': 381, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:19:40.473854+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:19:40.473854+0000', 'last_peered': '2023-12-12T02:19:40.473854+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:19:40.473854+0000', 'last_undegraded': '2023-12-12T02:19:40.473854+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.675 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:36.675 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 529, 'reported_epoch': 381, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:19:40.469829+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:19:40.469829+0000', 'last_peered': '2023-12-12T02:19:40.469829+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:19:40.469829+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:36.675 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:36.984 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:36.985 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6/20 objects degraded (30.000%), 5 pgs degraded, 49 pgs undersized","count":54},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2246,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":44},{"state_name":"active+clean","count":24},{"state_name":"active+undersized+degraded","count":5}],"num_pgs":73,"num_pools":3,"num_objects":10,"data_bytes":590387,"bytes_used":3068669952,"bytes_avail":286841622528,"bytes_total":289910292480,"degraded_objects":6,"degraded_total":20,"degraded_ratio":0.29999999999999999},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":39,"modified":"2023-12-12T02:19:58.417704+0000","services":{"rbd-mirror":{"daemons":{"summary":"","4553":{"start_epoch":14,"start_stamp":"2023-12-12T01:52:52.065321+0000","gid":4553,"addr":"172.21.15.140:0/1248111478","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)","ceph_version_short":"19.0.0-130-g3d2db44f","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi140","id":"mirror.3","instance_id":"4553","kernel_description":"#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023","kernel_version":"5.15.0-91-generic","mem_swap_kb":"2097148","mem_total_kb":"32746956","os":"Linux"},"task_status":{}},"5651":{"start_epoch":26,"start_stamp":"2023-12-12T02:08:15.105162+0000","gid":5651,"addr":"172.21.15.140:0/313796890","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)","ceph_version_short":"19.0.0-130-g3d2db44f","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi140","id":"mirror.2","instance_id":"5651","kernel_description":"#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023","kernel_version":"5.15.0-91-generic","mem_swap_kb":"2097148","mem_total_kb":"32746956","os":"Linux"},"task_status":{}},"6604":{"start_epoch":37,"start_stamp":"2023-12-12T02:17:20.255525+0000","gid":6604,"addr":"172.21.15.140:0/3076915661","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)","ceph_version_short":"19.0.0-130-g3d2db44f","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi140","id":"mirror.0","instance_id":"6604","kernel_description":"#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023","kernel_version":"5.15.0-91-generic","mem_swap_kb":"2097148","mem_total_kb":"32746956","os":"Linux"},"task_status":{}},"6772":{"start_epoch":39,"start_stamp":"2023-12-12T02:19:58.235977+0000","gid":6772,"addr":"172.21.15.140:0/1700115751","metadata":{"arch":"x86_64","ceph_release":"reef","ceph_version":"ceph version 19.0.0-130-g3d2db44f (3d2db44fa991bc355074aa9b7647f63b77201fef) reef (dev)","ceph_version_short":"19.0.0-130-g3d2db44f","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"ubuntu","distro_description":"Ubuntu 22.04.1 LTS","distro_version":"22.04","hostname":"smithi140","id":"mirror.1","instance_id":"6772","kernel_description":"#101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023","kernel_version":"5.15.0-91-generic","mem_swap_kb":"2097148","mem_total_kb":"32746956","os":"Linux"},"task_status":{}}}}}},"progress_events":{}} 2023-12-12T02:20:37.002 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:37.002 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:37.278 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:37.278 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:37.294 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1499,"stamp":"2023-12-12T02:20:36.443738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":238,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3014804,"kb_used_data":369816,"kb_used_omap":291,"kb_used_meta":2644636,"kb_avail":280100716,"statfs":{"total":289910292480,"available":286823133184,"internally_reserved":0,"allocated":378691584,"data_stored":374183005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2708107818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":217,"apply_latency_ms":217,"commit_latency_ns":217000000,"apply_latency_ns":217000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536833078,"num_objects":-164,"num_object_clones":0,"num_object_copies":-328,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-164,"num_whiteouts":0,"num_read":-113602,"num_read_kb":-39868702,"num_write":-397940,"num_write_kb":-144697292,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-3,"num_bytes_recovered":-8388608,"num_keys_recovered":-2,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.005075"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.592626+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:19:51.592626+0000","last_peered":"2023-12-12T02:19:51.592626+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:19:51.592626+0000","last_undegraded":"2023-12-12T02:19:51.592626+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":399,"reported_epoch":382,"state":"active+undersized","last_fresh":"2023-12-12T02:19:51.593769+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:19:51.593769+0000","last_peered":"2023-12-12T02:19:51.593769+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:19:51.593769+0000","last_undegraded":"2023-12-12T02:19:51.593769+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471006+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:19:40.471006+0000","last_peered":"2023-12-12T02:19:40.471006+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:19:40.471006+0000","last_undegraded":"2023-12-12T02:19:40.471006+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":399,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.471106+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:19:40.471106+0000","last_peered":"2023-12-12T02:19:40.471106+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:19:40.471106+0000","last_undegraded":"2023-12-12T02:19:40.471106+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":381,"state":"active+undersized","last_fresh":"2023-12-12T02:19:40.473854+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:19:40.473854+0000","last_peered":"2023-12-12T02:19:40.473854+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:19:40.473854+0000","last_undegraded":"2023-12-12T02:19:40.473854+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":395,"reported_epoch":381,"state":"active+clean","last_fresh":"2023-12-12T02:19:40.473378+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:19:40.473378+0000","last_peered":"2023-12-12T02:19:40.473378+0000","last_clean":"2023-12-12T02:19:40.473378+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:19:40.473378+0000","last_undegraded":"2023-12-12T02:19:40.473378+0000","last_fullsized":"2023-12-12T02:19:40.473378+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":529,"reported_epoch":381,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:19:40.469829+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:19:40.469829+0000","last_peered":"2023-12-12T02:19:40.469829+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:19:40.469829+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750608,"kb_used_data":3080,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621232,"statfs":{"total":96636764160,"available":95868141568,"internally_reserved":0,"allocated":3153920,"data_stored":1688322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":204,"apply_latency_ms":204,"commit_latency_ns":204000000,"apply_latency_ns":204000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738819,"num_pgs":98,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877576,"kb_used_data":3648,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494264,"statfs":{"total":96636764160,"available":95738126336,"internally_reserved":0,"allocated":3735552,"data_stored":2274510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:38.488 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~200s 2023-12-12T02:20:40.296 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:40.593 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:40.593 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:40.610 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2023-12-12T02:20:40.445004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":229,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3014828,"kb_used_data":369840,"kb_used_omap":291,"kb_used_meta":2644636,"kb_avail":280100692,"statfs":{"total":289910292480,"available":286823108608,"internally_reserved":0,"allocated":378716160,"data_stored":374190891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2708107818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":434,"apply_latency_ms":434,"commit_latency_ns":434000000,"apply_latency_ns":434000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.941140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750608,"kb_used_data":3080,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621232,"statfs":{"total":96636764160,"available":95868141568,"internally_reserved":0,"allocated":3153920,"data_stored":1688322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":204,"apply_latency_ms":204,"commit_latency_ns":204000000,"apply_latency_ns":204000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877600,"kb_used_data":3672,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494240,"statfs":{"total":96636764160,"available":95738101760,"internally_reserved":0,"allocated":3760128,"data_stored":2282396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":222,"apply_latency_ms":222,"commit_latency_ns":222000000,"apply_latency_ns":222000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:40.611 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:40.898 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:40.899 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:40.918 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2023-12-12T02:20:40.445004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":229,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3014828,"kb_used_data":369840,"kb_used_omap":291,"kb_used_meta":2644636,"kb_avail":280100692,"statfs":{"total":289910292480,"available":286823108608,"internally_reserved":0,"allocated":378716160,"data_stored":374190891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2708107818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":434,"apply_latency_ms":434,"commit_latency_ns":434000000,"apply_latency_ns":434000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.941140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750608,"kb_used_data":3080,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621232,"statfs":{"total":96636764160,"available":95868141568,"internally_reserved":0,"allocated":3153920,"data_stored":1688322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":204,"apply_latency_ms":204,"commit_latency_ns":204000000,"apply_latency_ns":204000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877600,"kb_used_data":3672,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494240,"statfs":{"total":96636764160,"available":95738101760,"internally_reserved":0,"allocated":3760128,"data_stored":2282396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":222,"apply_latency_ms":222,"commit_latency_ns":222000000,"apply_latency_ns":222000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:40.919 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.920 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:40.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.920 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:40.920 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:40.920 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:41.261 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:41.261 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2251,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3087183872,"bytes_avail":286823108608,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:20:41.281 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:41.281 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:41.561 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:41.562 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:41.579 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1501,"stamp":"2023-12-12T02:20:40.445004+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":229,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3014828,"kb_used_data":369840,"kb_used_omap":291,"kb_used_meta":2644636,"kb_avail":280100692,"statfs":{"total":289910292480,"available":286823108608,"internally_reserved":0,"allocated":378716160,"data_stored":374190891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2708107818},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":434,"apply_latency_ms":434,"commit_latency_ns":434000000,"apply_latency_ns":434000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.941140"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750608,"kb_used_data":3080,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621232,"statfs":{"total":96636764160,"available":95868141568,"internally_reserved":0,"allocated":3153920,"data_stored":1688322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":204,"apply_latency_ms":204,"commit_latency_ns":204000000,"apply_latency_ns":204000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.40600000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":877600,"kb_used_data":3672,"kb_used_omap":86,"kb_used_meta":873833,"kb_avail":93494240,"statfs":{"total":96636764160,"available":95738101760,"internally_reserved":0,"allocated":3760128,"data_stored":2282396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":894805503},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":222,"apply_latency_ms":222,"commit_latency_ns":222000000,"apply_latency_ns":222000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:19:33 2023","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.53500000000000003,"15min":0.53700000000000003},"min":{"1min":0.17000000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64800000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.61399999999999999,"15min":0.59799999999999998},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.1220000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.31}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:43.892 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~205s 2023-12-12T02:20:44.581 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:44.890 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:44.890 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:44.908 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1503,"stamp":"2023-12-12T02:20:44.446225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.59099999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:44.909 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:45.192 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:45.192 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:45.209 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1503,"stamp":"2023-12-12T02:20:44.446225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.59099999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:45.210 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:45.210 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.210 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:45.210 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:45.211 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:45.211 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:45.527 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:45.527 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2255,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:20:45.545 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:45.545 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:45.808 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:45.809 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:45.825 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1503,"stamp":"2023-12-12T02:20:44.446225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2108,"num_read_kb":-2001,"num_write":-407,"num_write_kb":-316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.59099999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:48.827 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:49.127 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:49.127 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:49.145 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1505,"stamp":"2023-12-12T02:20:48.447452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.501},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:49.146 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:49.295 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~211s 2023-12-12T02:20:49.414 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:49.415 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:49.431 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1505,"stamp":"2023-12-12T02:20:48.447452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.501},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:49.432 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.433 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:49.433 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:49.433 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:49.746 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:49.746 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2259,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:20:49.764 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:49.764 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:50.029 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:50.029 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:50.045 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1505,"stamp":"2023-12-12T02:20:48.447452+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003714"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738823,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.20000000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.255}]}]},{"osd":1,"up_from":8,"seq":34359738822,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.501},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:53.047 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:53.345 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:53.346 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:53.363 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1507,"stamp":"2023-12-12T02:20:52.448731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:53.364 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:53.654 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:53.655 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:53.673 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1507,"stamp":"2023-12-12T02:20:52.448731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:53.674 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.675 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:53.675 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:53.675 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:54.004 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:54.004 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2264,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:20:54.022 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:54.022 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:54.289 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:54.290 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:54.306 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1507,"stamp":"2023-12-12T02:20:52.448731+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577331+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:31.577331+0000","last_peered":"2023-12-12T02:20:31.577331+0000","last_clean":"2023-12-12T02:20:31.577331+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:31.577331+0000","last_undegraded":"2023-12-12T02:20:31.577331+0000","last_fullsized":"2023-12-12T02:20:31.577331+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738824,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.498}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:54.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~216s 2023-12-12T02:20:57.308 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:57.614 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:57.614 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:57.631 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1509,"stamp":"2023-12-12T02:20:56.449981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:57.632 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:57.893 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:57.893 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:57.911 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1509,"stamp":"2023-12-12T02:20:56.449981+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738823,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35899999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:20:57.912 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:20:57.913 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:20:57.913 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:20:58.234 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:58.235 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2268,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:20:58.252 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:20:58.252 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:20:58.519 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:20:58.520 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:20:58.536 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1510,"stamp":"2023-12-12T02:20:58.450615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003772"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:00.102 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~222s 2023-12-12T02:21:01.537 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:01.834 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:01.835 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:01.852 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1511,"stamp":"2023-12-12T02:21:00.451263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:01.853 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:02.122 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:02.122 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:02.138 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1511,"stamp":"2023-12-12T02:21:00.451263+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003811"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":397,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.518368+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:31.518368+0000","last_peered":"2023-12-12T02:20:31.518368+0000","last_clean":"2023-12-12T02:20:31.518368+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:31.518368+0000","last_undegraded":"2023-12-12T02:20:31.518368+0000","last_fullsized":"2023-12-12T02:20:31.518368+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738825,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.61699999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738824,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.50900000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:02.139 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:02.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.140 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:02.140 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:02.140 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:02.449 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:02.449 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2272,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:02.466 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:02.467 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:02.745 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:02.745 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:02.763 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1512,"stamp":"2023-12-12T02:21:02.451892+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003767"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:05.507 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~227s 2023-12-12T02:21:05.765 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:06.052 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:06.052 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:06.068 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1513,"stamp":"2023-12-12T02:21:04.452529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:06.069 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:06.357 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:06.357 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:06.375 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1513,"stamp":"2023-12-12T02:21:04.452529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003797"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.510692+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:31.510692+0000', 'last_peered': '2023-12-12T02:20:31.510692+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:31.510692+0000', 'last_undegraded': '2023-12-12T02:20:31.510692+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.518139+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:31.518139+0000', 'last_peered': '2023-12-12T02:20:31.518139+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:31.518139+0000', 'last_undegraded': '2023-12-12T02:20:31.518139+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:06.376 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.377 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:06.377 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:06.377 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:06.694 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:06.694 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2276,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:06.711 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:06.711 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:06.999 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:06.999 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:07.016 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1514,"stamp":"2023-12-12T02:21:06.453142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003755"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.510692+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:31.510692+0000","last_peered":"2023-12-12T02:20:31.510692+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:31.510692+0000","last_undegraded":"2023-12-12T02:20:31.510692+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.518139+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:31.518139+0000","last_peered":"2023-12-12T02:20:31.518139+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:31.518139+0000","last_undegraded":"2023-12-12T02:20:31.518139+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.55600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:10.018 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:10.319 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:10.320 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:10.338 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1515,"stamp":"2023-12-12T02:21:08.453763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003781"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.504}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:10.339 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:10.623 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:10.624 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:10.640 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1516,"stamp":"2023-12-12T02:21:10.454396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003780"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.504}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:10.641 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:10.641 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:10.642 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:10.643 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:10.910 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~232s 2023-12-12T02:21:10.957 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:10.957 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2280,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:10.975 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:10.975 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:11.246 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:11.246 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:11.263 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1516,"stamp":"2023-12-12T02:21:10.454396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003780"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738827,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66600000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738826,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.504}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:14.265 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:14.554 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:14.555 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:14.571 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2023-12-12T02:21:14.455680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003787"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.63700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:14.572 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:14.858 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:14.859 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:14.877 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2023-12-12T02:21:14.455680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003787"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.63700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:14.878 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:14.879 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:14.879 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:15.205 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:15.205 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2285,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:15.223 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:15.223 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:15.502 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:15.503 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:15.520 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1518,"stamp":"2023-12-12T02:21:14.455680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003787"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738828,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.63700000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738827,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.57999999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:16.314 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~238s 2023-12-12T02:21:18.522 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:18.819 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:18.820 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:18.838 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1520,"stamp":"2023-12-12T02:21:18.456894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.371},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.437},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.5}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:18.839 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:19.127 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:19.127 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:19.145 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1520,"stamp":"2023-12-12T02:21:18.456894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.371},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.437},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.5}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:19.146 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:19.147 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:19.147 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:19.465 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:19.465 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2289,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:19.483 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:19.483 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:19.766 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:19.766 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:19.786 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1520,"stamp":"2023-12-12T02:21:18.456894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003751"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738829,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.371},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.45200000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738828,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.437},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.5}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:21.717 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~243s 2023-12-12T02:21:22.788 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:23.087 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:23.088 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:23.104 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2023-12-12T02:21:22.458117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:23.105 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:23.392 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:23.393 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:23.410 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2023-12-12T02:21:22.458117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.411 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:23.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:23.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.412 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:23.412 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:23.412 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:23.740 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:23.740 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2293,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:23.758 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:23.758 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:24.032 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:24.033 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:24.049 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1522,"stamp":"2023-12-12T02:21:22.458117+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003721"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738830,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.66500000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:27.051 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:27.121 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~249s 2023-12-12T02:21:27.344 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:27.345 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:27.361 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1524,"stamp":"2023-12-12T02:21:26.459412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:27.362 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:27.625 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:27.625 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:27.642 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1524,"stamp":"2023-12-12T02:21:26.459412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:27.643 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.644 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:27.644 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:27.644 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:27.977 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:27.977 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2297,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:27.995 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:27.995 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:28.270 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:28.271 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:28.288 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1524,"stamp":"2023-12-12T02:21:26.459412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003732"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738829,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:31.290 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:31.595 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:31.595 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:31.613 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1526,"stamp":"2023-12-12T02:21:30.460706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.65400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:31.614 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:31.910 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:31.910 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:31.928 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1526,"stamp":"2023-12-12T02:21:30.460706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":18,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.531460+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:31.531460+0000","last_peered":"2023-12-12T02:20:31.531460+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:31.531460+0000","last_undegraded":"2023-12-12T02:20:31.531460+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":531,"reported_epoch":383,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:31.512776+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:31.512776+0000","last_peered":"2023-12-12T02:20:31.512776+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:31.512776+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738831,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.68200000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738830,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.65400000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:31.929 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:31.929 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.929 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:31.929 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 18, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.531460+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:31.531460+0000', 'last_peered': '2023-12-12T02:20:31.531460+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:31.531460+0000', 'last_undegraded': '2023-12-12T02:20:31.531460+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:31.930 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 531, 'reported_epoch': 383, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:31.512776+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:31.512776+0000', 'last_peered': '2023-12-12T02:20:31.512776+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:31.512776+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:31.930 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:32.255 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:32.255 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2302,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:32.273 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:32.273 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:32.525 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~254s 2023-12-12T02:21:32.541 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:32.542 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:32.558 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1527,"stamp":"2023-12-12T02:21:32.461322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003776"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.44},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:35.560 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:35.857 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:35.857 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:35.875 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1528,"stamp":"2023-12-12T02:21:34.461989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.44},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:35.876 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:36.168 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:36.169 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:36.187 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1528,"stamp":"2023-12-12T02:21:34.461989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003871"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":398,"reported_epoch":383,"state":"active+clean","last_fresh":"2023-12-12T02:20:31.577747+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:31.577747+0000","last_peered":"2023-12-12T02:20:31.577747+0000","last_clean":"2023-12-12T02:20:31.577747+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:31.577747+0000","last_undegraded":"2023-12-12T02:20:31.577747+0000","last_fullsized":"2023-12-12T02:20:31.577747+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738832,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.44},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58599999999999997}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:36.188 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.189 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:36.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:36.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:36.190 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:36.511 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:36.512 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2306,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:36.529 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:36.529 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:36.806 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:36.806 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:36.822 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1529,"stamp":"2023-12-12T02:21:36.462591+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003790"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.56699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738831,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.49099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:37.929 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~259s 2023-12-12T02:21:39.824 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:40.126 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:40.127 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:40.144 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1530,"stamp":"2023-12-12T02:21:38.463208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003796"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.56699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:40.146 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:40.433 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:40.434 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:40.451 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1530,"stamp":"2023-12-12T02:21:38.463208+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003796"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.56699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.452 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:40.453 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:40.453 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:40.789 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:40.789 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2310,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:40.807 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:40.807 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:41.092 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:41.093 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:41.109 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1531,"stamp":"2023-12-12T02:21:40.463845+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003761"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738833,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:20:35 2023","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.57199999999999995,"15min":0.56999999999999995},"min":{"1min":0.16200000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":7.4450000000000003,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.68400000000000005,"15min":0.68200000000000005},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":7.5490000000000004,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.56699999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738832,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:20:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.52500000000000002,"15min":0.53200000000000003},"min":{"1min":0.22700000000000001,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.64900000000000002,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.58899999999999997,"15min":0.59099999999999997},"min":{"1min":0.26300000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.65200000000000002,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52600000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:43.333 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~265s 2023-12-12T02:21:44.111 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:44.393 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:44.393 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:44.409 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2023-12-12T02:21:42.464425+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003718"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.65500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:44.410 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:44.691 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:44.691 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:44.709 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1533,"stamp":"2023-12-12T02:21:44.465053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.65500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:44.710 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:44.711 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:44.711 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:45.049 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:45.050 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2315,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:45.068 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:45.068 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:45.333 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:45.333 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:45.350 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1533,"stamp":"2023-12-12T02:21:44.465053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003731"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738834,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.65500000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738833,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:48.352 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:48.642 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:48.642 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:48.659 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2023-12-12T02:21:48.466247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:48.660 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:48.738 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~270s 2023-12-12T02:21:48.921 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:48.921 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:48.937 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2023-12-12T02:21:48.466247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:48.938 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:48.938 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.938 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:48.939 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:48.940 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:49.277 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:49.277 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2319,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:49.297 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:49.297 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:49.567 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:49.567 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:49.583 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1535,"stamp":"2023-12-12T02:21:48.466247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738835,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.30099999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738834,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:52.585 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:52.885 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:52.886 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:52.902 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2023-12-12T02:21:52.467487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003642"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:52.903 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:53.187 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:53.187 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:53.205 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2023-12-12T02:21:52.467487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003642"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:53.206 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:53.206 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.206 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:53.206 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:53.207 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:53.207 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:53.526 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:53.526 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2323,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:53.544 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:53.544 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:53.819 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:53.820 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:53.836 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1537,"stamp":"2023-12-12T02:21:52.467487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003642"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738836,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.58299999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:54.142 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~276s 2023-12-12T02:21:56.837 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:57.125 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:57.125 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:57.141 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2023-12-12T02:21:56.468748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:57.142 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:57.424 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:57.424 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:57.442 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2023-12-12T02:21:56.468748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:21:57.443 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.444 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:21:57.444 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.444 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:21:57.444 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:21:57.444 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:21:57.788 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:57.789 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2327,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:21:57.807 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:21:57.807 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:21:58.069 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:21:58.070 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:21:58.086 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2023-12-12T02:21:56.468748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738835,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.48699999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:21:59.546 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~281s 2023-12-12T02:22:01.089 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:01.383 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:01.383 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:01.399 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1541,"stamp":"2023-12-12T02:22:00.470003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.625}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:01.400 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:01.675 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:01.675 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:01.691 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1541,"stamp":"2023-12-12T02:22:00.470003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.625}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:01.691 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.692 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:22:01.693 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:01.693 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:22:02.036 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:02.036 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2332,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:22:02.055 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:22:02.055 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:02.323 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:02.324 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:02.340 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1541,"stamp":"2023-12-12T02:22:00.470003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003756"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738837,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.54000000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738836,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.625}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:04.951 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~286s 2023-12-12T02:22:05.342 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:05.632 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:05.633 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:05.649 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1543,"stamp":"2023-12-12T02:22:04.471337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003850"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.434},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.51500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.496}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:05.650 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:05.933 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:05.933 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:05.951 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1543,"stamp":"2023-12-12T02:22:04.471337+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003850"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738838,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.434},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.51500000000000001}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.496}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:05.952 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:22:05.952 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.952 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:22:05.952 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:22:05.953 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:05.953 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:22:06.286 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:06.286 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2336,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:22:06.304 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:22:06.304 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:06.568 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:06.569 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:06.585 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1544,"stamp":"2023-12-12T02:22:06.471957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003818"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738837,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.496}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:09.587 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:09.883 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:09.883 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:09.901 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1545,"stamp":"2023-12-12T02:22:08.472588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:09.905 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:10.173 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:10.173 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:10.189 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1545,"stamp":"2023-12-12T02:22:08.472588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003840"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.190 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:22:10.191 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.191 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:22:10.191 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.191 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:22:10.191 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:10.191 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:22:10.355 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~292s 2023-12-12T02:22:10.525 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:10.525 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2340,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:22:10.543 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:22:10.543 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:10.815 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:10.816 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:10.831 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1546,"stamp":"2023-12-12T02:22:10.473248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003881"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738839,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.57999999999999996}]}]},{"osd":1,"up_from":8,"seq":34359738838,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.35999999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:13.833 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:14.130 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:14.131 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:14.148 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1547,"stamp":"2023-12-12T02:22:12.473839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003835"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:14.149 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:14.437 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:14.438 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:14.455 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1547,"stamp":"2023-12-12T02:22:12.473839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003835"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:14.456 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:22:14.456 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:22:14.457 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:14.458 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:22:14.784 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:14.784 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2344,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:22:14.801 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:22:14.801 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:15.066 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:15.066 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:15.083 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1548,"stamp":"2023-12-12T02:22:14.474511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003816"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738840,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.55600000000000005}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:15.759 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~297s 2023-12-12T02:22:18.085 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:18.385 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:18.385 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:18.403 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1549,"stamp":"2023-12-12T02:22:16.475095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003758"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.315},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.41199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738839,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.53800000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:18.404 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:18.699 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:18.700 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:18.718 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1550,"stamp":"2023-12-12T02:22:18.475723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.315},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.41199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.442},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.7 is not active+clean 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.7', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.523218+0000', 'last_change': '2023-12-12T02:17:19.452325+0000', 'last_active': '2023-12-12T02:20:31.523218+0000', 'last_peered': '2023-12-12T02:20:31.523218+0000', 'last_clean': '2023-12-12T02:17:16.148578+0000', 'last_became_active': '2023-12-12T02:17:19.452325+0000', 'last_became_peered': '2023-12-12T02:17:19.452325+0000', 'last_unstale': '2023-12-12T02:20:31.523218+0000', 'last_undegraded': '2023-12-12T02:20:31.523218+0000', 'last_fullsized': '2023-12-12T02:17:19.419474+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.6 is not active+clean 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.6', 'version': "0'0", 'reported_seq': 401, 'reported_epoch': 383, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:31.522814+0000', 'last_change': '2023-12-12T02:17:19.447110+0000', 'last_active': '2023-12-12T02:20:31.522814+0000', 'last_peered': '2023-12-12T02:20:31.522814+0000', 'last_clean': '2023-12-12T02:17:16.159550+0000', 'last_became_active': '2023-12-12T02:17:19.447110+0000', 'last_became_peered': '2023-12-12T02:17:19.447110+0000', 'last_unstale': '2023-12-12T02:20:31.522814+0000', 'last_undegraded': '2023-12-12T02:20:31.522814+0000', 'last_fullsized': '2023-12-12T02:17:19.413486+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.5 is not active+clean 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.5', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.669694+0000', 'last_change': '2023-12-12T02:17:19.445257+0000', 'last_active': '2023-12-12T02:20:34.669694+0000', 'last_peered': '2023-12-12T02:20:34.669694+0000', 'last_clean': '2023-12-12T02:17:16.148203+0000', 'last_became_active': '2023-12-12T02:17:19.445257+0000', 'last_became_peered': '2023-12-12T02:17:19.445257+0000', 'last_unstale': '2023-12-12T02:20:34.669694+0000', 'last_undegraded': '2023-12-12T02:20:34.669694+0000', 'last_fullsized': '2023-12-12T02:17:19.410969+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.4 is not active+clean 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 402, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:34.942068+0000', 'last_change': '2023-12-12T02:17:19.443840+0000', 'last_active': '2023-12-12T02:20:34.942068+0000', 'last_peered': '2023-12-12T02:20:34.942068+0000', 'last_clean': '2023-12-12T02:17:16.152998+0000', 'last_became_active': '2023-12-12T02:17:19.443840+0000', 'last_became_peered': '2023-12-12T02:17:19.443840+0000', 'last_unstale': '2023-12-12T02:20:34.942068+0000', 'last_undegraded': '2023-12-12T02:20:34.942068+0000', 'last_fullsized': '2023-12-12T02:17:19.413284+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:PG 2.2 is not active+clean 2023-12-12T02:22:18.719 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 19, 'reported_epoch': 384, 'state': 'active+undersized', 'last_fresh': '2023-12-12T02:20:36.469550+0000', 'last_change': '2023-12-12T02:17:19.460993+0000', 'last_active': '2023-12-12T02:20:36.469550+0000', 'last_peered': '2023-12-12T02:20:36.469550+0000', 'last_clean': '2023-12-12T01:43:26.020059+0000', 'last_became_active': '2023-12-12T02:17:19.460993+0000', 'last_became_peered': '2023-12-12T02:17:19.460993+0000', 'last_unstale': '2023-12-12T02:20:36.469550+0000', 'last_undegraded': '2023-12-12T02:20:36.469550+0000', 'last_fullsized': '2023-12-12T02:17:19.417000+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 13, 'last_epoch_clean': 14, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:26.020059+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.720 INFO:tasks.ceph.ceph_manager.cluster2:PG 1.0 is not active+clean 2023-12-12T02:22:18.720 INFO:tasks.ceph.ceph_manager.cluster2:{'pgid': '1.0', 'version': "11'97", 'reported_seq': 532, 'reported_epoch': 384, 'state': 'active+undersized+degraded', 'last_fresh': '2023-12-12T02:20:34.944001+0000', 'last_change': '2023-12-12T02:17:19.467284+0000', 'last_active': '2023-12-12T02:20:34.944001+0000', 'last_peered': '2023-12-12T02:20:34.944001+0000', 'last_clean': '2023-12-12T02:17:16.149082+0000', 'last_became_active': '2023-12-12T02:17:19.467284+0000', 'last_became_peered': '2023-12-12T02:17:19.467284+0000', 'last_unstale': '2023-12-12T02:20:34.944001+0000', 'last_undegraded': '2023-12-12T02:17:19.410574+0000', 'last_fullsized': '2023-12-12T02:17:19.410278+0000', 'mapping_epoch': 374, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 10, 'last_epoch_clean': 11, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'last_clean_scrub_stamp': '2023-12-12T01:43:21.993354+0000', 'objects_scrubbed': 0, 'log_size': 97, 'log_dups_size': 0, 'ondisk_log_size': 97, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 590368, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 4, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 2, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 106, 'num_read_kb': 91, 'num_write': 137, 'num_write_kb': 1894, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1], 'acting': [1], 'avail_no_missing': ['1'], 'object_location_counts': [{'shards': '1', 'objects': 2}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-12-12T02:22:18.720 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 status --format=json 2023-12-12T02:22:19.064 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:19.064 INFO:teuthology.orchestra.run.smithi140.stdout:{"fsid":"59e1e678-6e7f-4060-9001-c278538e473e","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2/4 objects degraded (50.000%), 1 pg degraded, 6 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_age":2349,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":1},"osdmap":{"epoch":384,"num_osds":3,"num_up_osds":2,"osd_up_since":1702347438,"num_in_osds":3,"osd_in_since":1702345390,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+undersized","count":5},{"state_name":"active+clean","count":3},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":9,"num_pools":2,"num_objects":2,"data_bytes":590368,"bytes_used":3105673216,"bytes_avail":286804619264,"bytes_total":289910292480,"degraded_objects":2,"degraded_total":4,"degraded_ratio":0.5},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":40,"modified":"2023-12-12T02:20:36.443787+0000","services":{}},"progress_events":{}} 2023-12-12T02:22:19.084 INFO:tasks.ceph.ceph_manager.cluster2:no progress seen, keeping timeout for now 2023-12-12T02:22:19.084 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:22:19.371 INFO:teuthology.orchestra.run.smithi140.stdout: 2023-12-12T02:22:19.371 INFO:teuthology.orchestra.run.smithi140.stderr:dumped all 2023-12-12T02:22:19.387 INFO:teuthology.orchestra.run.smithi140.stdout:{"pg_ready":true,"pg_map":{"version":1550,"stamp":"2023-12-12T02:22:18.475723+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":12,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":108,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3032884,"kb_used_data":369844,"kb_used_omap":291,"kb_used_meta":2662684,"kb_avail":280082636,"statfs":{"total":289910292480,"available":286804619264,"internally_reserved":0,"allocated":378720256,"data_stored":374194559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":298454,"internal_metadata":2726588970},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003766"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.523218+0000","last_change":"2023-12-12T02:17:19.452325+0000","last_active":"2023-12-12T02:20:31.523218+0000","last_peered":"2023-12-12T02:20:31.523218+0000","last_clean":"2023-12-12T02:17:16.148578+0000","last_became_active":"2023-12-12T02:17:19.452325+0000","last_became_peered":"2023-12-12T02:17:19.452325+0000","last_unstale":"2023-12-12T02:20:31.523218+0000","last_undegraded":"2023-12-12T02:20:31.523218+0000","last_fullsized":"2023-12-12T02:17:19.419474+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:13:33.421102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":401,"reported_epoch":383,"state":"active+undersized","last_fresh":"2023-12-12T02:20:31.522814+0000","last_change":"2023-12-12T02:17:19.447110+0000","last_active":"2023-12-12T02:20:31.522814+0000","last_peered":"2023-12-12T02:20:31.522814+0000","last_clean":"2023-12-12T02:17:16.159550+0000","last_became_active":"2023-12-12T02:17:19.447110+0000","last_became_peered":"2023-12-12T02:17:19.447110+0000","last_unstale":"2023-12-12T02:20:31.522814+0000","last_undegraded":"2023-12-12T02:20:31.522814+0000","last_fullsized":"2023-12-12T02:17:19.413486+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:11:40.969664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.669694+0000","last_change":"2023-12-12T02:17:19.445257+0000","last_active":"2023-12-12T02:20:34.669694+0000","last_peered":"2023-12-12T02:20:34.669694+0000","last_clean":"2023-12-12T02:17:16.148203+0000","last_became_active":"2023-12-12T02:17:19.445257+0000","last_became_peered":"2023-12-12T02:17:19.445257+0000","last_unstale":"2023-12-12T02:20:34.669694+0000","last_undegraded":"2023-12-12T02:20:34.669694+0000","last_fullsized":"2023-12-12T02:17:19.410969+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:46:23.229319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":402,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:34.942068+0000","last_change":"2023-12-12T02:17:19.443840+0000","last_active":"2023-12-12T02:20:34.942068+0000","last_peered":"2023-12-12T02:20:34.942068+0000","last_clean":"2023-12-12T02:17:16.152998+0000","last_became_active":"2023-12-12T02:17:19.443840+0000","last_became_peered":"2023-12-12T02:17:19.443840+0000","last_unstale":"2023-12-12T02:20:34.942068+0000","last_undegraded":"2023-12-12T02:20:34.942068+0000","last_fullsized":"2023-12-12T02:17:19.413284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:50.424306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":19,"reported_epoch":384,"state":"active+undersized","last_fresh":"2023-12-12T02:20:36.469550+0000","last_change":"2023-12-12T02:17:19.460993+0000","last_active":"2023-12-12T02:20:36.469550+0000","last_peered":"2023-12-12T02:20:36.469550+0000","last_clean":"2023-12-12T01:43:26.020059+0000","last_became_active":"2023-12-12T02:17:19.460993+0000","last_became_peered":"2023-12-12T02:17:19.460993+0000","last_unstale":"2023-12-12T02:20:36.469550+0000","last_undegraded":"2023-12-12T02:20:36.469550+0000","last_fullsized":"2023-12-12T02:17:19.417000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:32:45.911234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.347379+0000","last_change":"2023-12-12T01:43:27.051738+0000","last_active":"2023-12-12T02:20:36.347379+0000","last_peered":"2023-12-12T02:20:36.347379+0000","last_clean":"2023-12-12T02:20:36.347379+0000","last_became_active":"2023-12-12T01:43:27.050866+0000","last_became_peered":"2023-12-12T01:43:27.050866+0000","last_unstale":"2023-12-12T02:20:36.347379+0000","last_undegraded":"2023-12-12T02:20:36.347379+0000","last_fullsized":"2023-12-12T02:20:36.347379+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:08:48.233614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":399,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:36.332011+0000","last_change":"2023-12-12T01:43:27.051340+0000","last_active":"2023-12-12T02:20:36.332011+0000","last_peered":"2023-12-12T02:20:36.332011+0000","last_clean":"2023-12-12T02:20:36.332011+0000","last_became_active":"2023-12-12T01:43:27.049750+0000","last_became_peered":"2023-12-12T01:43:27.049750+0000","last_unstale":"2023-12-12T02:20:36.332011+0000","last_undegraded":"2023-12-12T02:20:36.332011+0000","last_fullsized":"2023-12-12T02:20:36.332011+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:15:16.095155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":384,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.942685+0000","last_change":"2023-12-12T01:43:27.051630+0000","last_active":"2023-12-12T02:20:34.942685+0000","last_peered":"2023-12-12T02:20:34.942685+0000","last_clean":"2023-12-12T02:20:34.942685+0000","last_became_active":"2023-12-12T01:43:27.051010+0000","last_became_peered":"2023-12-12T01:43:27.051010+0000","last_unstale":"2023-12-12T02:20:34.942685+0000","last_undegraded":"2023-12-12T02:20:34.942685+0000","last_fullsized":"2023-12-12T02:20:34.942685+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:26.020059+0000","last_clean_scrub_stamp":"2023-12-12T01:43:26.020059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:06:24.221159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":532,"reported_epoch":384,"state":"active+undersized+degraded","last_fresh":"2023-12-12T02:20:34.944001+0000","last_change":"2023-12-12T02:17:19.467284+0000","last_active":"2023-12-12T02:20:34.944001+0000","last_peered":"2023-12-12T02:20:34.944001+0000","last_clean":"2023-12-12T02:17:16.149082+0000","last_became_active":"2023-12-12T02:17:19.467284+0000","last_became_peered":"2023-12-12T02:17:19.467284+0000","last_unstale":"2023-12-12T02:20:34.944001+0000","last_undegraded":"2023-12-12T02:17:19.410574+0000","last_fullsized":"2023-12-12T02:17:19.410278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":10,"last_epoch_clean":11,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:43:21.993354+0000","last_clean_scrub_stamp":"2023-12-12T01:43:21.993354+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:15:28.537745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":11,"acting":11,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":1,"acting":1,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738841,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":750620,"kb_used_data":3088,"kb_used_omap":99,"kb_used_meta":747420,"kb_avail":93621220,"statfs":{"total":96636764160,"available":95868129280,"internally_reserved":0,"allocated":3162112,"data_stored":1692009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":102240,"internal_metadata":765358240},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.48099999999999998,"15min":0.68799999999999994},"min":{"1min":0.193,"5min":0.151,"15min":0.151},"max":{"1min":2.464,"5min":2.464,"15min":19.268999999999998},"last":0.157},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59099999999999997,"15min":0.57299999999999995},"min":{"1min":0.26500000000000001,"5min":0.214,"15min":0.17899999999999999},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.9790000000000001},"last":0.22800000000000001}]},{"osd":1,"last update":"Tue Dec 12 02:21:37 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.20000000000000001,"5min":0.16200000000000001,"15min":0.16200000000000001},"max":{"1min":0.874,"5min":7.4450000000000003,"15min":7.4450000000000003},"last":0.315},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.64000000000000001,"15min":0.68700000000000006},"min":{"1min":0.255,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.97499999999999998,"5min":7.5490000000000004,"15min":7.5490000000000004},"last":0.41199999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738840,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":895644,"kb_used_data":3668,"kb_used_omap":86,"kb_used_meta":891881,"kb_avail":93476196,"statfs":{"total":96636764160,"available":95719624704,"internally_reserved":0,"allocated":3756032,"data_stored":2282377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":88577,"internal_metadata":913286655},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:16:29 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51200000000000001,"15min":0.79600000000000004},"min":{"1min":0.17899999999999999,"5min":0.17599999999999999,"15min":0.157},"max":{"1min":1.024,"5min":2.984,"15min":19.609999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.62,"15min":0.753},"min":{"1min":0.20599999999999999,"5min":0.19900000000000001,"15min":0.189},"max":{"1min":0.68200000000000005,"5min":3.8130000000000002,"15min":17.244},"last":0.81699999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:21:39 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.30399999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.57799999999999996,"5min":3.7730000000000001,"15min":4.4530000000000003},"last":0.442},{"interface":"front","average":{"1min":0.505,"5min":0.57399999999999995,"15min":0.58299999999999996},"min":{"1min":0.27400000000000002,"5min":0.187,"15min":0.187},"max":{"1min":0.65400000000000003,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.54100000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738781,"num_pgs":96,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1386620,"kb_used_data":363088,"kb_used_omap":105,"kb_used_meta":1023382,"kb_avail":92985220,"statfs":{"total":96636764160,"available":95216865280,"internally_reserved":0,"allocated":371802112,"data_stored":370220173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107637,"internal_metadata":1047944075},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.20300000000000001,"5min":0.17799999999999999,"15min":0.17499999999999999},"max":{"1min":0.65200000000000002,"5min":2.6850000000000001,"15min":7.8849999999999998},"last":0.17299999999999999},{"interface":"front","average":{"1min":1.1779999999999999,"5min":0.97099999999999997,"15min":0.93799999999999994},"min":{"1min":0.29399999999999998,"5min":0.21099999999999999,"15min":0.17100000000000001},"max":{"1min":8.109,"5min":9.8450000000000006,"15min":27.541},"last":0.21299999999999999}]},{"osd":2,"last update":"Tue Dec 12 02:16:28 2023","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.94599999999999995,"15min":0.72699999999999998},"min":{"1min":0.25800000000000001,"5min":0.23000000000000001,"15min":0.19700000000000001},"max":{"1min":8.2129999999999992,"5min":9.7810000000000006,"15min":10.029},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":1.1659999999999999,"15min":0.84099999999999997},"min":{"1min":0.27700000000000002,"5min":0.189,"15min":0.189},"max":{"1min":0.74099999999999999,"5min":22.98,"15min":22.98},"last":0.20399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:22:21.162 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~303s 2023-12-12T02:22:21.163 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2023-12-12T02:22:21.567 INFO:tasks.ceph.cluster2.osd.1:Sent signal 15 2023-12-12T02:22:21.567 INFO:tasks.ceph.cluster2.osd.2:Sent signal 15 2023-12-12T02:22:21.567 INFO:tasks.ceph.cluster2.mon.a:Sent signal 15 2023-12-12T02:22:21.567 INFO:tasks.ceph.cluster2.mgr.x:Sent signal 15 2023-12-12T02:22:21.568 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T02:22:21.567+0000 7f7681447640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 (PID: 34903) UID: 0 2023-12-12T02:22:21.568 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T02:22:21.567+0000 7f7681447640 -1 osd.1 384 *** Got signal Terminated *** 2023-12-12T02:22:21.568 INFO:tasks.ceph.cluster2.osd.1.smithi140.stderr:2023-12-12T02:22:21.567+0000 7f7681447640 -1 osd.1 384 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:22:21.567+0000 7fd7b93c2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 (PID: 34906) UID: 0 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:22:21.567+0000 7fd7b93c2640 -1 osd.2 384 *** Got signal Terminated *** 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.osd.2.smithi140.stderr:2023-12-12T02:22:21.567+0000 7fd7b93c2640 -1 osd.2 384 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.mon.a.smithi140.stderr:2023-12-12T02:22:21.567+0000 7f6ebc8cd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mon -f --cluster cluster2 -i a (PID: 34756) UID: 0 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.mon.a.smithi140.stderr:2023-12-12T02:22:21.567+0000 7f6ebc8cd640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T02:22:21.570+0000 7fb59b9c7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-mgr -f --cluster cluster2 -i x (PID: 39502) UID: 0 2023-12-12T02:22:21.569 INFO:tasks.ceph.cluster2.mgr.x.smithi055.stderr:2023-12-12T02:22:21.570+0000 7fb59b9c7640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-12-12T02:22:22.389 DEBUG:teuthology.orchestra.run.smithi140:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2023-12-12T02:24:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 124 2023-12-12T02:24:22.413 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-12T02:24:22.415 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-12T02:24:22.415 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-12T02:24:22.416 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2023-12-12T02:24:22.416 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:24:22.416 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-12T02:24:22.416 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 5: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0' 2023-12-12T02:24:22.416 INFO:tasks.ceph.cluster2.osd.0:Stopped 2023-12-12T02:24:22.417 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2023-12-12T02:24:22.417 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:24:22.417 INFO:tasks.ceph.cluster2.osd.1:Stopped 2023-12-12T02:24:22.417 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2023-12-12T02:24:22.417 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:24:22.417 INFO:tasks.ceph.cluster2.osd.2:Stopped 2023-12-12T02:24:22.417 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-12T02:24:22.418 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2023-12-12T02:24:22.418 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:24:22.418 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2023-12-12T02:24:22.418 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-12T02:24:22.418 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2023-12-12T02:24:22.418 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:24:22.418 INFO:tasks.ceph.cluster2.mon.a:Stopped 2023-12-12T02:24:22.418 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-12T02:24:22.419 DEBUG:teuthology.orchestra.run.smithi140:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-12T02:24:22.434 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T02:24:22.434 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2023-12-12T02:24:22.797 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T02:24:22.797 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2023-12-12T02:24:23.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T02:24:23.078 DEBUG:teuthology.orchestra.run.smithi140:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2023-12-12T02:24:23.433 INFO:tasks.ceph:Archiving mon data... 2023-12-12T02:24:23.434 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/data/cluster2.mon.a.tgz 2023-12-12T02:24:23.434 DEBUG:teuthology.orchestra.run.smithi140:> mktemp 2023-12-12T02:24:23.439 INFO:teuthology.orchestra.run.smithi140.stdout:/tmp/tmp.UyZg992FLt 2023-12-12T02:24:23.439 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.UyZg992FLt 2023-12-12T02:24:23.586 DEBUG:teuthology.orchestra.run.smithi140:> sudo chmod 0666 /tmp/tmp.UyZg992FLt 2023-12-12T02:24:23.651 DEBUG:teuthology.orchestra.remote:smithi140:/tmp/tmp.UyZg992FLt is 400KB 2023-12-12T02:24:23.713 DEBUG:teuthology.orchestra.run.smithi140:> rm -fr /tmp/tmp.UyZg992FLt 2023-12-12T02:24:23.718 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-12T02:24:23.719 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-12T02:24:23.721 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2023-12-12T02:24:23.775 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-12T02:24:23.776 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-12-12T02:24:23.788 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2023-12-12T02:24:23.788 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:24:24.091 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:24.091 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:24:24.109 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2023-12-12T02:24:23.922221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286076,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829444,"statfs":{"total":289910292480,"available":289617350656,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228225+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T02:21:48.228225+0000","last_peered":"2023-12-12T02:21:48.228225+0000","last_clean":"2023-12-12T02:21:48.228225+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:21:48.228225+0000","last_undegraded":"2023-12-12T02:21:48.228225+0000","last_fullsized":"2023-12-12T02:21:48.228225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:46.969394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228197+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T02:21:48.228197+0000","last_peered":"2023-12-12T02:21:48.228197+0000","last_clean":"2023-12-12T02:21:48.228197+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:21:48.228197+0000","last_undegraded":"2023-12-12T02:21:48.228197+0000","last_fullsized":"2023-12-12T02:21:48.228197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:02:20.609969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.245759+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T02:20:34.245759+0000","last_peered":"2023-12-12T02:20:34.245759+0000","last_clean":"2023-12-12T02:20:34.245759+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:20:34.245759+0000","last_undegraded":"2023-12-12T02:20:34.245759+0000","last_fullsized":"2023-12-12T02:20:34.245759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:21:00.854399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.244568+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T02:20:34.244568+0000","last_peered":"2023-12-12T02:20:34.244568+0000","last_clean":"2023-12-12T02:20:34.244568+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:20:34.244568+0000","last_undegraded":"2023-12-12T02:20:34.244568+0000","last_fullsized":"2023-12-12T02:20:34.244568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:14.937456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":473,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:33.446718+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T02:20:33.446718+0000","last_peered":"2023-12-12T02:20:33.446718+0000","last_clean":"2023-12-12T02:20:33.446718+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:20:33.446718+0000","last_undegraded":"2023-12-12T02:20:33.446718+0000","last_fullsized":"2023-12-12T02:20:33.446718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:58:21.541457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.252803+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T02:20:34.252803+0000","last_peered":"2023-12-12T02:20:34.252803+0000","last_clean":"2023-12-12T02:20:34.252803+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:20:34.252803+0000","last_undegraded":"2023-12-12T02:20:34.252803+0000","last_fullsized":"2023-12-12T02:20:34.252803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.517709+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T02:20:53.517709+0000","last_peered":"2023-12-12T02:20:53.517709+0000","last_clean":"2023-12-12T02:20:53.517709+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:20:53.517709+0000","last_undegraded":"2023-12-12T02:20:53.517709+0000","last_fullsized":"2023-12-12T02:20:53.517709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.247277+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T02:20:34.247277+0000","last_peered":"2023-12-12T02:20:34.247277+0000","last_clean":"2023-12-12T02:20:34.247277+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:20:34.247277+0000","last_undegraded":"2023-12-12T02:20:34.247277+0000","last_fullsized":"2023-12-12T02:20:34.247277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:48:36.680390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.219443+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T02:20:53.219443+0000","last_peered":"2023-12-12T02:20:53.219443+0000","last_clean":"2023-12-12T02:20:53.219443+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:20:53.219443+0000","last_undegraded":"2023-12-12T02:20:53.219443+0000","last_fullsized":"2023-12-12T02:20:53.219443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93116,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278724,"statfs":{"total":96636764160,"available":96541413376,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.216},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57699999999999996,"15min":0.60099999999999998},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.72099999999999997,"5min":0.82099999999999995,"15min":5.8079999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59399999999999997,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":13.891},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.71899999999999997},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.31},"max":{"1min":0.79300000000000004,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.74399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89536,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282304,"statfs":{"total":96636764160,"available":96545079296,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103424,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268416,"statfs":{"total":96636764160,"available":96530857984,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.184},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":3.004},"last":0.433},{"interface":"front","average":{"1min":0.625,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.20200000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":7.3109999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.214},"max":{"1min":0.70599999999999996,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69599999999999995,"15min":0.69699999999999995},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.27000000000000002},"max":{"1min":0.84899999999999998,"5min":1.3009999999999999,"15min":7.6660000000000004},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:24:24.110 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:24:24.395 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:24.395 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:24:24.412 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2023-12-12T02:24:23.922221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286076,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829444,"statfs":{"total":289910292480,"available":289617350656,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228225+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T02:21:48.228225+0000","last_peered":"2023-12-12T02:21:48.228225+0000","last_clean":"2023-12-12T02:21:48.228225+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:21:48.228225+0000","last_undegraded":"2023-12-12T02:21:48.228225+0000","last_fullsized":"2023-12-12T02:21:48.228225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:46.969394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228197+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T02:21:48.228197+0000","last_peered":"2023-12-12T02:21:48.228197+0000","last_clean":"2023-12-12T02:21:48.228197+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:21:48.228197+0000","last_undegraded":"2023-12-12T02:21:48.228197+0000","last_fullsized":"2023-12-12T02:21:48.228197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:02:20.609969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.245759+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T02:20:34.245759+0000","last_peered":"2023-12-12T02:20:34.245759+0000","last_clean":"2023-12-12T02:20:34.245759+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:20:34.245759+0000","last_undegraded":"2023-12-12T02:20:34.245759+0000","last_fullsized":"2023-12-12T02:20:34.245759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:21:00.854399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.244568+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T02:20:34.244568+0000","last_peered":"2023-12-12T02:20:34.244568+0000","last_clean":"2023-12-12T02:20:34.244568+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:20:34.244568+0000","last_undegraded":"2023-12-12T02:20:34.244568+0000","last_fullsized":"2023-12-12T02:20:34.244568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:14.937456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":473,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:33.446718+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T02:20:33.446718+0000","last_peered":"2023-12-12T02:20:33.446718+0000","last_clean":"2023-12-12T02:20:33.446718+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:20:33.446718+0000","last_undegraded":"2023-12-12T02:20:33.446718+0000","last_fullsized":"2023-12-12T02:20:33.446718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:58:21.541457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.252803+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T02:20:34.252803+0000","last_peered":"2023-12-12T02:20:34.252803+0000","last_clean":"2023-12-12T02:20:34.252803+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:20:34.252803+0000","last_undegraded":"2023-12-12T02:20:34.252803+0000","last_fullsized":"2023-12-12T02:20:34.252803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.517709+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T02:20:53.517709+0000","last_peered":"2023-12-12T02:20:53.517709+0000","last_clean":"2023-12-12T02:20:53.517709+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:20:53.517709+0000","last_undegraded":"2023-12-12T02:20:53.517709+0000","last_fullsized":"2023-12-12T02:20:53.517709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.247277+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T02:20:34.247277+0000","last_peered":"2023-12-12T02:20:34.247277+0000","last_clean":"2023-12-12T02:20:34.247277+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:20:34.247277+0000","last_undegraded":"2023-12-12T02:20:34.247277+0000","last_fullsized":"2023-12-12T02:20:34.247277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:48:36.680390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.219443+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T02:20:53.219443+0000","last_peered":"2023-12-12T02:20:53.219443+0000","last_clean":"2023-12-12T02:20:53.219443+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:20:53.219443+0000","last_undegraded":"2023-12-12T02:20:53.219443+0000","last_fullsized":"2023-12-12T02:20:53.219443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93116,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278724,"statfs":{"total":96636764160,"available":96541413376,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.216},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57699999999999996,"15min":0.60099999999999998},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.72099999999999997,"5min":0.82099999999999995,"15min":5.8079999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59399999999999997,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":13.891},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.71899999999999997},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.31},"max":{"1min":0.79300000000000004,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.74399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89536,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282304,"statfs":{"total":96636764160,"available":96545079296,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103424,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268416,"statfs":{"total":96636764160,"available":96530857984,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.184},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":3.004},"last":0.433},{"interface":"front","average":{"1min":0.625,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.20200000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":7.3109999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.214},"max":{"1min":0.70599999999999996,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69599999999999995,"15min":0.69699999999999995},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.27000000000000002},"max":{"1min":0.84899999999999998,"5min":1.3009999999999999,"15min":7.6660000000000004},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:24:24.413 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2023-12-12T02:24:24.413 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:24:24.686 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:24.686 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:24:24.702 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1708,"stamp":"2023-12-12T02:24:23.922221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286076,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829444,"statfs":{"total":289910292480,"available":289617350656,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002986"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228225+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T02:21:48.228225+0000","last_peered":"2023-12-12T02:21:48.228225+0000","last_clean":"2023-12-12T02:21:48.228225+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:21:48.228225+0000","last_undegraded":"2023-12-12T02:21:48.228225+0000","last_fullsized":"2023-12-12T02:21:48.228225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:46.969394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228197+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T02:21:48.228197+0000","last_peered":"2023-12-12T02:21:48.228197+0000","last_clean":"2023-12-12T02:21:48.228197+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:21:48.228197+0000","last_undegraded":"2023-12-12T02:21:48.228197+0000","last_fullsized":"2023-12-12T02:21:48.228197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:02:20.609969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.245759+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T02:20:34.245759+0000","last_peered":"2023-12-12T02:20:34.245759+0000","last_clean":"2023-12-12T02:20:34.245759+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:20:34.245759+0000","last_undegraded":"2023-12-12T02:20:34.245759+0000","last_fullsized":"2023-12-12T02:20:34.245759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:21:00.854399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.244568+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T02:20:34.244568+0000","last_peered":"2023-12-12T02:20:34.244568+0000","last_clean":"2023-12-12T02:20:34.244568+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:20:34.244568+0000","last_undegraded":"2023-12-12T02:20:34.244568+0000","last_fullsized":"2023-12-12T02:20:34.244568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:14.937456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":473,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:33.446718+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T02:20:33.446718+0000","last_peered":"2023-12-12T02:20:33.446718+0000","last_clean":"2023-12-12T02:20:33.446718+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:20:33.446718+0000","last_undegraded":"2023-12-12T02:20:33.446718+0000","last_fullsized":"2023-12-12T02:20:33.446718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:58:21.541457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.252803+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T02:20:34.252803+0000","last_peered":"2023-12-12T02:20:34.252803+0000","last_clean":"2023-12-12T02:20:34.252803+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:20:34.252803+0000","last_undegraded":"2023-12-12T02:20:34.252803+0000","last_fullsized":"2023-12-12T02:20:34.252803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.517709+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T02:20:53.517709+0000","last_peered":"2023-12-12T02:20:53.517709+0000","last_clean":"2023-12-12T02:20:53.517709+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:20:53.517709+0000","last_undegraded":"2023-12-12T02:20:53.517709+0000","last_fullsized":"2023-12-12T02:20:53.517709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.247277+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T02:20:34.247277+0000","last_peered":"2023-12-12T02:20:34.247277+0000","last_clean":"2023-12-12T02:20:34.247277+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:20:34.247277+0000","last_undegraded":"2023-12-12T02:20:34.247277+0000","last_fullsized":"2023-12-12T02:20:34.247277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:48:36.680390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.219443+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T02:20:53.219443+0000","last_peered":"2023-12-12T02:20:53.219443+0000","last_clean":"2023-12-12T02:20:53.219443+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:20:53.219443+0000","last_undegraded":"2023-12-12T02:20:53.219443+0000","last_fullsized":"2023-12-12T02:20:53.219443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93116,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278724,"statfs":{"total":96636764160,"available":96541413376,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.216},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57699999999999996,"15min":0.60099999999999998},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.72099999999999997,"5min":0.82099999999999995,"15min":5.8079999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59399999999999997,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":13.891},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.71899999999999997},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.31},"max":{"1min":0.79300000000000004,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.74399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89536,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282304,"statfs":{"total":96636764160,"available":96545079296,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103424,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268416,"statfs":{"total":96636764160,"available":96530857984,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.184},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":3.004},"last":0.433},{"interface":"front","average":{"1min":0.625,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.20200000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":7.3109999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.214},"max":{"1min":0.70599999999999996,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69599999999999995,"15min":0.69699999999999995},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.27000000000000002},"max":{"1min":0.84899999999999998,"5min":1.3009999999999999,"15min":7.6660000000000004},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:24:24.703 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2023-12-12T02:24:24.982 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:24.983 INFO:teuthology.orchestra.run.smithi055.stdout:{"epoch":460,"fsid":"f2278db0-8e6c-4cd2-a0a7-022133e15780","created":"2023-12-12T01:42:31.931335+0000","modified":"2023-12-12T02:20:33.433054+0000","last_up_change":"2023-12-12T01:42:42.145792+0000","last_in_change":"2023-12-12T01:42:32.893837+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-12-12T01:42:43.067634+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-12-12T01:42:47.590187+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9f50009f-ef20-4a77-9de9-cd9707279076","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6800","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6801","nonce":218205279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6802","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6803","nonce":218205279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6806","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6807","nonce":218205279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6804","nonce":218205279},{"type":"v1","addr":"172.21.15.55:6805","nonce":218205279}]},"public_addr":"172.21.15.55:6801/218205279","cluster_addr":"172.21.15.55:6803/218205279","heartbeat_back_addr":"172.21.15.55:6807/218205279","heartbeat_front_addr":"172.21.15.55:6805/218205279","state":["exists","up"]},{"osd":1,"uuid":"4ac775af-1ed3-4717-9081-30a58cc61291","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6816","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6817","nonce":624247608}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6818","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6819","nonce":624247608}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6822","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6823","nonce":624247608}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6820","nonce":624247608},{"type":"v1","addr":"172.21.15.55:6821","nonce":624247608}]},"public_addr":"172.21.15.55:6817/624247608","cluster_addr":"172.21.15.55:6819/624247608","heartbeat_back_addr":"172.21.15.55:6823/624247608","heartbeat_front_addr":"172.21.15.55:6821/624247608","state":["exists","up"]},{"osd":2,"uuid":"e7149351-6dc2-4e78-83d3-e37f097e7d60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":25,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6808","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6809","nonce":859848655}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6810","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6811","nonce":859848655}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6814","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6815","nonce":859848655}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.55:6812","nonce":859848655},{"type":"v1","addr":"172.21.15.55:6813","nonce":859848655}]},"public_addr":"172.21.15.55:6809/859848655","cluster_addr":"172.21.15.55:6811/859848655","heartbeat_back_addr":"172.21.15.55:6815/859848655","heartbeat_front_addr":"172.21.15.55:6813/859848655","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.157627+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.162269+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-12-12T01:42:38.509259+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-12-12T02:24:25.999 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2023-12-12T02:24:25.999 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-12-12T02:24:26.162 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-12T02:24:26.163 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-12T02:24:26.163 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-12T02:24:26.178 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2023-12-12T02:24:26.455 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 0 to deep-scrub 2023-12-12T02:24:26.472 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2023-12-12T02:24:26.473 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-12-12T02:24:26.624 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-12T02:24:26.624 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-12T02:24:26.625 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-12T02:24:26.641 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2023-12-12T02:24:26.927 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 1 to deep-scrub 2023-12-12T02:24:26.946 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2023-12-12T02:24:26.946 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-12-12T02:24:27.103 INFO:teuthology.orchestra.run.smithi055.stdout:{ 2023-12-12T02:24:27.103 INFO:teuthology.orchestra.run.smithi055.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2023-12-12T02:24:27.103 INFO:teuthology.orchestra.run.smithi055.stdout:} 2023-12-12T02:24:27.118 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2023-12-12T02:24:27.397 INFO:teuthology.orchestra.run.smithi055.stderr:instructed osd(s) 2 to deep-scrub 2023-12-12T02:24:27.414 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:24:27.701 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:27.702 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:24:27.720 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1709,"stamp":"2023-12-12T02:24:25.922729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286076,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829444,"statfs":{"total":289910292480,"available":289617350656,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003003"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228225+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T02:21:48.228225+0000","last_peered":"2023-12-12T02:21:48.228225+0000","last_clean":"2023-12-12T02:21:48.228225+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:21:48.228225+0000","last_undegraded":"2023-12-12T02:21:48.228225+0000","last_fullsized":"2023-12-12T02:21:48.228225+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:30:46.969394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:21:48.228197+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T02:21:48.228197+0000","last_peered":"2023-12-12T02:21:48.228197+0000","last_clean":"2023-12-12T02:21:48.228197+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:21:48.228197+0000","last_undegraded":"2023-12-12T02:21:48.228197+0000","last_fullsized":"2023-12-12T02:21:48.228197+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:02:20.609969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.245759+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T02:20:34.245759+0000","last_peered":"2023-12-12T02:20:34.245759+0000","last_clean":"2023-12-12T02:20:34.245759+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:20:34.245759+0000","last_undegraded":"2023-12-12T02:20:34.245759+0000","last_fullsized":"2023-12-12T02:20:34.245759+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T06:21:00.854399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.244568+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T02:20:34.244568+0000","last_peered":"2023-12-12T02:20:34.244568+0000","last_clean":"2023-12-12T02:20:34.244568+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:20:34.244568+0000","last_undegraded":"2023-12-12T02:20:34.244568+0000","last_fullsized":"2023-12-12T02:20:34.244568+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:51:14.937456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":473,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:33.446718+0000","last_change":"2023-12-12T01:42:48.909922+0000","last_active":"2023-12-12T02:20:33.446718+0000","last_peered":"2023-12-12T02:20:33.446718+0000","last_clean":"2023-12-12T02:20:33.446718+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:20:33.446718+0000","last_undegraded":"2023-12-12T02:20:33.446718+0000","last_fullsized":"2023-12-12T02:20:33.446718+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T03:58:21.541457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.252803+0000","last_change":"2023-12-12T01:42:48.905185+0000","last_active":"2023-12-12T02:20:34.252803+0000","last_peered":"2023-12-12T02:20:34.252803+0000","last_clean":"2023-12-12T02:20:34.252803+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:20:34.252803+0000","last_undegraded":"2023-12-12T02:20:34.252803+0000","last_fullsized":"2023-12-12T02:20:34.252803+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:51:33.271182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.517709+0000","last_change":"2023-12-12T01:42:48.903852+0000","last_active":"2023-12-12T02:20:53.517709+0000","last_peered":"2023-12-12T02:20:53.517709+0000","last_clean":"2023-12-12T02:20:53.517709+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:20:53.517709+0000","last_undegraded":"2023-12-12T02:20:53.517709+0000","last_fullsized":"2023-12-12T02:20:53.517709+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T08:38:11.073230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":476,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:34.247277+0000","last_change":"2023-12-12T01:42:48.905755+0000","last_active":"2023-12-12T02:20:34.247277+0000","last_peered":"2023-12-12T02:20:34.247277+0000","last_clean":"2023-12-12T02:20:34.247277+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:20:34.247277+0000","last_undegraded":"2023-12-12T02:20:34.247277+0000","last_fullsized":"2023-12-12T02:20:34.247277+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T07:48:36.680390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":607,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:20:53.219443+0000","last_change":"2023-12-12T01:42:44.177833+0000","last_active":"2023-12-12T02:20:53.219443+0000","last_peered":"2023-12-12T02:20:53.219443+0000","last_clean":"2023-12-12T02:20:53.219443+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:20:53.219443+0000","last_undegraded":"2023-12-12T02:20:53.219443+0000","last_fullsized":"2023-12-12T02:20:53.219443+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:43.147672+0000","last_clean_scrub_stamp":"2023-12-12T01:42:43.147672+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:03:58.114924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738873,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93116,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278724,"statfs":{"total":96636764160,"available":96541413376,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.216},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57699999999999996,"15min":0.60099999999999998},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.72099999999999997,"5min":0.82099999999999995,"15min":5.8079999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59399999999999997,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":13.891},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.71899999999999997},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.31},"max":{"1min":0.79300000000000004,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.74399999999999999}]}]},{"osd":1,"up_from":8,"seq":34359738871,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89536,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282304,"statfs":{"total":96636764160,"available":96545079296,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.68999999999999995}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.58799999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738873,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103424,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268416,"statfs":{"total":96636764160,"available":96530857984,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.184},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":3.004},"last":0.433},{"interface":"front","average":{"1min":0.625,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.20200000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":7.3109999999999999},"last":0.56299999999999994}]},{"osd":2,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.214},"max":{"1min":0.70599999999999996,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69599999999999995,"15min":0.69699999999999995},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.27000000000000002},"max":{"1min":0.84899999999999998,"5min":1.3009999999999999,"15min":7.6660000000000004},"last":0.66600000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:24:27.723 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.723 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-12-12T01:42:43.147672+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=43, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:27.724 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-12T02:24:47.725 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:24:48.033 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:24:48.034 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:24:48.053 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1720,"stamp":"2023-12-12T02:24:47.928229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286064,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829456,"statfs":{"total":289910292480,"available":289617362944,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002958"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:23:58.248732+0000","last_change":"2023-12-12T01:42:48.902776+0000","last_active":"2023-12-12T02:23:58.248732+0000","last_peered":"2023-12-12T02:23:58.248732+0000","last_clean":"2023-12-12T02:23:58.248732+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:23:58.248732+0000","last_undegraded":"2023-12-12T02:23:58.248732+0000","last_fullsized":"2023-12-12T02:23:58.248732+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:23:58.248675+0000","last_change":"2023-12-12T01:42:48.911170+0000","last_active":"2023-12-12T02:23:58.248675+0000","last_peered":"2023-12-12T02:23:58.248675+0000","last_clean":"2023-12-12T02:23:58.248675+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:23:58.248675+0000","last_undegraded":"2023-12-12T02:23:58.248675+0000","last_fullsized":"2023-12-12T02:23:58.248675+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:26.931429+0000","last_change":"2023-12-12T01:42:48.911476+0000","last_active":"2023-12-12T02:24:26.931429+0000","last_peered":"2023-12-12T02:24:26.931429+0000","last_clean":"2023-12-12T02:24:26.931429+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:24:26.931429+0000","last_undegraded":"2023-12-12T02:24:26.931429+0000","last_fullsized":"2023-12-12T02:24:26.931429+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":478,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:26.931322+0000","last_change":"2023-12-12T01:42:48.908778+0000","last_active":"2023-12-12T02:24:26.931322+0000","last_peered":"2023-12-12T02:24:26.931322+0000","last_clean":"2023-12-12T02:24:26.931322+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:24:26.931322+0000","last_undegraded":"2023-12-12T02:24:26.931322+0000","last_fullsized":"2023-12-12T02:24:26.931322+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T01:42:47.871831+0000","last_clean_scrub_stamp":"2023-12-12T01:42:47.871831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":483,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:28.180221+0000","last_change":"2023-12-12T02:24:28.180221+0000","last_active":"2023-12-12T02:24:28.180221+0000","last_peered":"2023-12-12T02:24:28.180221+0000","last_clean":"2023-12-12T02:24:28.180221+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:24:28.180221+0000","last_undegraded":"2023-12-12T02:24:28.180221+0000","last_fullsized":"2023-12-12T02:24:28.180221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:28.180021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:28.180021+0000","last_clean_scrub_stamp":"2023-12-12T02:24:28.180021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:19:19.242593+0000","scrub_duration":0.068201208999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:29.438524+0000","last_change":"2023-12-12T02:24:29.438547+0000","last_active":"2023-12-12T02:24:29.438524+0000","last_peered":"2023-12-12T02:24:29.438524+0000","last_clean":"2023-12-12T02:24:29.438524+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:24:29.438524+0000","last_undegraded":"2023-12-12T02:24:29.438524+0000","last_fullsized":"2023-12-12T02:24:29.438524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:29.438407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:29.438407+0000","last_clean_scrub_stamp":"2023-12-12T02:24:29.438407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:25:54.423992+0000","scrub_duration":0.15586191799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:28.444982+0000","last_change":"2023-12-12T02:24:28.445005+0000","last_active":"2023-12-12T02:24:28.444982+0000","last_peered":"2023-12-12T02:24:28.444982+0000","last_clean":"2023-12-12T02:24:28.444982+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:24:28.444982+0000","last_undegraded":"2023-12-12T02:24:28.444982+0000","last_fullsized":"2023-12-12T02:24:28.444982+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:28.444867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:28.444867+0000","last_clean_scrub_stamp":"2023-12-12T02:24:28.444867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T13:21:06.872285+0000","scrub_duration":0.13205800100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:40.547475+0000","last_change":"2023-12-12T02:24:40.547501+0000","last_active":"2023-12-12T02:24:40.547475+0000","last_peered":"2023-12-12T02:24:40.547475+0000","last_clean":"2023-12-12T02:24:40.547475+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:24:40.547475+0000","last_undegraded":"2023-12-12T02:24:40.547475+0000","last_fullsized":"2023-12-12T02:24:40.547475+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:40.547331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:40.547331+0000","last_clean_scrub_stamp":"2023-12-12T02:24:40.547331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T11:00:44.371591+0000","scrub_duration":0.21460769199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"460'99","reported_seq":618,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:36.330764+0000","last_change":"2023-12-12T02:24:36.330832+0000","last_active":"2023-12-12T02:24:36.330764+0000","last_peered":"2023-12-12T02:24:36.330764+0000","last_clean":"2023-12-12T02:24:36.330764+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:24:36.330764+0000","last_undegraded":"2023-12-12T02:24:36.330764+0000","last_fullsized":"2023-12-12T02:24:36.330764+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'99","last_scrub_stamp":"2023-12-12T02:24:36.330536+0000","last_deep_scrub":"460'99","last_deep_scrub_stamp":"2023-12-12T02:24:36.330536+0000","last_clean_scrub_stamp":"2023-12-12T02:24:36.330536+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:45:05.255673+0000","scrub_duration":0.011063668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738877,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93112,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278728,"statfs":{"total":96636764160,"available":96541417472,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55700000000000005,"15min":0.56000000000000005},"min":{"1min":0.44500000000000001,"5min":0.442,"15min":0.216},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57699999999999996,"15min":0.60099999999999998},"min":{"1min":0.48299999999999998,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.72099999999999997,"5min":0.82099999999999995,"15min":5.8079999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Tue Dec 12 02:23:56 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59399999999999997,"15min":0.628},"min":{"1min":0.48799999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.77100000000000002,"5min":0.78900000000000003,"15min":13.891},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.68899999999999995,"15min":0.71899999999999997},"min":{"1min":0.61499999999999999,"5min":0.54900000000000004,"15min":0.31},"max":{"1min":0.79300000000000004,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.67100000000000004}]}]},{"osd":1,"up_from":8,"seq":34359738875,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89532,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282308,"statfs":{"total":96636764160,"available":96545083392,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.64200000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738877,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103420,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268420,"statfs":{"total":96636764160,"available":96530862080,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.51700000000000002,"15min":0.55200000000000005},"min":{"1min":0.45900000000000002,"5min":0.41099999999999998,"15min":0.184},"max":{"1min":0.63100000000000001,"5min":0.79300000000000004,"15min":3.004},"last":0.496},{"interface":"front","average":{"1min":0.625,"5min":0.58999999999999997,"15min":0.61699999999999999},"min":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.20200000000000001},"max":{"1min":0.752,"5min":0.81100000000000005,"15min":7.3109999999999999},"last":0.59499999999999997}]},{"osd":2,"last update":"Tue Dec 12 02:23:45 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61799999999999999,"15min":0.63800000000000001},"min":{"1min":0.50800000000000001,"5min":0.505,"15min":0.214},"max":{"1min":0.70599999999999996,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.69599999999999995,"15min":0.69699999999999995},"min":{"1min":0.64100000000000001,"5min":0.58499999999999996,"15min":0.27000000000000002},"max":{"1min":0.84899999999999998,"5min":1.3009999999999999,"15min":7.6660000000000004},"last":0.66800000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:24:48.054 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:48.054 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:48.055 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:48.055 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-12-12T01:42:47.871831+0000 time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=1, tm_min=42, tm_sec=47, tm_wday=1, tm_yday=346, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=12, tm_mday=12, tm_hour=2, tm_min=24, tm_sec=24, tm_wday=1, tm_yday=346, tm_isdst=0) 2023-12-12T02:24:48.055 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-12-12T02:25:08.056 DEBUG:teuthology.orchestra.run.smithi055:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2023-12-12T02:25:08.363 INFO:teuthology.orchestra.run.smithi055.stdout: 2023-12-12T02:25:08.363 INFO:teuthology.orchestra.run.smithi055.stderr:dumped all 2023-12-12T02:25:08.381 INFO:teuthology.orchestra.run.smithi055.stdout:{"pg_ready":true,"pg_map":{"version":1730,"stamp":"2023-12-12T02:25:07.933232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":286064,"kb_used_data":12248,"kb_used_omap":4,"kb_used_meta":273787,"kb_avail":282829456,"statfs":{"total":289910292480,"available":289617362944,"internally_reserved":0,"allocated":12541952,"data_stored":7276901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":280358223},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002992"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":485,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:49.638958+0000","last_change":"2023-12-12T02:24:49.638980+0000","last_active":"2023-12-12T02:24:49.638958+0000","last_peered":"2023-12-12T02:24:49.638958+0000","last_clean":"2023-12-12T02:24:49.638958+0000","last_became_active":"2023-12-12T01:42:48.902142+0000","last_became_peered":"2023-12-12T01:42:48.902142+0000","last_unstale":"2023-12-12T02:24:49.638958+0000","last_undegraded":"2023-12-12T02:24:49.638958+0000","last_fullsized":"2023-12-12T02:24:49.638958+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:49.638842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:49.638842+0000","last_clean_scrub_stamp":"2023-12-12T02:24:49.638842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T05:09:11.294288+0000","scrub_duration":0.209956325,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":485,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:44.420768+0000","last_change":"2023-12-12T02:24:44.420821+0000","last_active":"2023-12-12T02:24:44.420768+0000","last_peered":"2023-12-12T02:24:44.420768+0000","last_clean":"2023-12-12T02:24:44.420768+0000","last_became_active":"2023-12-12T01:42:48.910408+0000","last_became_peered":"2023-12-12T01:42:48.910408+0000","last_unstale":"2023-12-12T02:24:44.420768+0000","last_undegraded":"2023-12-12T02:24:44.420768+0000","last_fullsized":"2023-12-12T02:24:44.420768+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:44.420519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:44.420519+0000","last_clean_scrub_stamp":"2023-12-12T02:24:44.420519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T11:40:23.604335+0000","scrub_duration":0.072167329000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:45.581435+0000","last_change":"2023-12-12T02:24:45.581459+0000","last_active":"2023-12-12T02:24:45.581435+0000","last_peered":"2023-12-12T02:24:45.581435+0000","last_clean":"2023-12-12T02:24:45.581435+0000","last_became_active":"2023-12-12T01:42:48.910487+0000","last_became_peered":"2023-12-12T01:42:48.910487+0000","last_unstale":"2023-12-12T02:24:45.581435+0000","last_undegraded":"2023-12-12T02:24:45.581435+0000","last_fullsized":"2023-12-12T02:24:45.581435+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:45.581309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:45.581309+0000","last_clean_scrub_stamp":"2023-12-12T02:24:45.581309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T04:45:19.185881+0000","scrub_duration":0.18912135899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:51.448021+0000","last_change":"2023-12-12T02:24:51.448072+0000","last_active":"2023-12-12T02:24:51.448021+0000","last_peered":"2023-12-12T02:24:51.448021+0000","last_clean":"2023-12-12T02:24:51.448021+0000","last_became_active":"2023-12-12T01:42:48.908305+0000","last_became_peered":"2023-12-12T01:42:48.908305+0000","last_unstale":"2023-12-12T02:24:51.448021+0000","last_undegraded":"2023-12-12T02:24:51.448021+0000","last_fullsized":"2023-12-12T02:24:51.448021+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:51.447804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:51.447804+0000","last_clean_scrub_stamp":"2023-12-12T02:24:51.447804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T02:28:46.264937+0000","scrub_duration":0.0032473319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":483,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:28.180221+0000","last_change":"2023-12-12T02:24:28.180221+0000","last_active":"2023-12-12T02:24:28.180221+0000","last_peered":"2023-12-12T02:24:28.180221+0000","last_clean":"2023-12-12T02:24:28.180221+0000","last_became_active":"2023-12-12T01:42:48.909500+0000","last_became_peered":"2023-12-12T01:42:48.909500+0000","last_unstale":"2023-12-12T02:24:28.180221+0000","last_undegraded":"2023-12-12T02:24:28.180221+0000","last_fullsized":"2023-12-12T02:24:28.180221+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:28.180021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:28.180021+0000","last_clean_scrub_stamp":"2023-12-12T02:24:28.180021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T12:19:19.242593+0000","scrub_duration":0.068201208999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:29.438524+0000","last_change":"2023-12-12T02:24:29.438547+0000","last_active":"2023-12-12T02:24:29.438524+0000","last_peered":"2023-12-12T02:24:29.438524+0000","last_clean":"2023-12-12T02:24:29.438524+0000","last_became_active":"2023-12-12T01:42:48.904552+0000","last_became_peered":"2023-12-12T01:42:48.904552+0000","last_unstale":"2023-12-12T02:24:29.438524+0000","last_undegraded":"2023-12-12T02:24:29.438524+0000","last_fullsized":"2023-12-12T02:24:29.438524+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:29.438407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:29.438407+0000","last_clean_scrub_stamp":"2023-12-12T02:24:29.438407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T10:25:54.423992+0000","scrub_duration":0.15586191799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":487,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:28.445005+0000","last_change":"2023-12-12T02:24:28.445005+0000","last_active":"2023-12-12T02:24:28.445005+0000","last_peered":"2023-12-12T02:24:28.445005+0000","last_clean":"2023-12-12T02:24:28.445005+0000","last_became_active":"2023-12-12T01:42:48.902977+0000","last_became_peered":"2023-12-12T01:42:48.902977+0000","last_unstale":"2023-12-12T02:24:28.445005+0000","last_undegraded":"2023-12-12T02:24:28.445005+0000","last_fullsized":"2023-12-12T02:24:28.445005+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:28.444867+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:28.444867+0000","last_clean_scrub_stamp":"2023-12-12T02:24:28.444867+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T13:21:06.872285+0000","scrub_duration":0.13205800100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":486,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:40.547475+0000","last_change":"2023-12-12T02:24:40.547501+0000","last_active":"2023-12-12T02:24:40.547475+0000","last_peered":"2023-12-12T02:24:40.547475+0000","last_clean":"2023-12-12T02:24:40.547475+0000","last_became_active":"2023-12-12T01:42:48.905097+0000","last_became_peered":"2023-12-12T01:42:48.905097+0000","last_unstale":"2023-12-12T02:24:40.547475+0000","last_undegraded":"2023-12-12T02:24:40.547475+0000","last_fullsized":"2023-12-12T02:24:40.547475+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-12-12T02:24:40.547331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-12-12T02:24:40.547331+0000","last_clean_scrub_stamp":"2023-12-12T02:24:40.547331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T11:00:44.371591+0000","scrub_duration":0.21460769199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"460'99","reported_seq":619,"reported_epoch":460,"state":"active+clean","last_fresh":"2023-12-12T02:24:36.330832+0000","last_change":"2023-12-12T02:24:36.330832+0000","last_active":"2023-12-12T02:24:36.330832+0000","last_peered":"2023-12-12T02:24:36.330832+0000","last_clean":"2023-12-12T02:24:36.330832+0000","last_became_active":"2023-12-12T01:42:44.175332+0000","last_became_peered":"2023-12-12T01:42:44.175332+0000","last_unstale":"2023-12-12T02:24:36.330832+0000","last_undegraded":"2023-12-12T02:24:36.330832+0000","last_fullsized":"2023-12-12T02:24:36.330832+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"460'99","last_scrub_stamp":"2023-12-12T02:24:36.330536+0000","last_deep_scrub":"460'99","last_deep_scrub_stamp":"2023-12-12T02:24:36.330536+0000","last_clean_scrub_stamp":"2023-12-12T02:24:36.330536+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-12-13T09:45:05.255673+0000","scrub_duration":0.011063668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738881,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":93112,"kb_used_data":3696,"kb_used_omap":1,"kb_used_meta":89406,"kb_avail":94278728,"statfs":{"total":96636764160,"available":96541417472,"internally_reserved":0,"allocated":3784704,"data_stored":2032055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91552197},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.216},"max":{"1min":0.76300000000000001,"5min":0.77000000000000002,"15min":4.8159999999999998},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.57099999999999995,"15min":0.60199999999999998},"min":{"1min":0.51900000000000002,"5min":0.43099999999999999,"15min":0.247},"max":{"1min":0.64800000000000002,"5min":0.72099999999999997,"15min":5.8079999999999998},"last":0.58599999999999997}]},{"osd":1,"last update":"Tue Dec 12 02:25:02 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.629},"min":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.27000000000000002},"max":{"1min":0.70999999999999996,"5min":0.77100000000000002,"15min":13.891},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.68200000000000005,"15min":0.71899999999999997},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.31},"max":{"1min":0.76200000000000001,"5min":0.79400000000000004,"15min":13.930999999999999},"last":0.69799999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738879,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89532,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282308,"statfs":{"total":96636764160,"available":96545083392,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.53700000000000003,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.23100000000000001},"max":{"1min":0.73899999999999999,"5min":0.75700000000000001,"15min":5.4930000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.69999999999999996,"15min":0.69399999999999995},"min":{"1min":0.57599999999999996,"5min":0.56000000000000005,"15min":0.36099999999999999},"max":{"1min":0.79100000000000004,"5min":0.91000000000000003,"15min":6.5899999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Tue Dec 12 02:24:04 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.54200000000000004,"15min":0.57499999999999996},"min":{"1min":0.433,"5min":0.433,"15min":0.219},"max":{"1min":0.69399999999999995,"5min":0.70399999999999996,"15min":4.3799999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.625,"15min":0.63800000000000001},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.27000000000000002},"max":{"1min":0.71399999999999997,"5min":1.266,"15min":5.5750000000000002},"last":0.505}]}]},{"osd":0,"up_from":8,"seq":34359738881,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":103420,"kb_used_data":4276,"kb_used_omap":1,"kb_used_meta":99134,"kb_avail":94268420,"statfs":{"total":96636764160,"available":96530862080,"internally_reserved":0,"allocated":4378624,"data_stored":2622423,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101513669},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Dec 12 02:24:49 2023","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.51400000000000001,"15min":0.54600000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.184},"max":{"1min":0.59099999999999997,"5min":0.79300000000000004,"15min":3.004},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.59799999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.20200000000000001},"max":{"1min":0.67200000000000004,"5min":0.81100000000000005,"15min":5.0830000000000002},"last":0.66300000000000003}]},{"osd":2,"last update":"Tue Dec 12 02:24:49 2023","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.59899999999999998,"15min":0.63400000000000001},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.214},"max":{"1min":0.78300000000000003,"5min":0.96899999999999997,"15min":3.0990000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.70399999999999996,"15min":0.67600000000000005},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.27000000000000002},"max":{"1min":0.76100000000000001,"5min":1.3009999999999999,"15min":2.8260000000000001},"last":0.72199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-12-12T02:25:08.383 DEBUG:teuthology.orchestra.run.smithi055:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2023-12-12T02:25:08.706 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1918, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-12T02:25:08.707 INFO:teuthology.misc:Shutting down mds daemons... 2023-12-12T02:25:08.707 INFO:teuthology.misc:Shutting down osd daemons... 2023-12-12T02:25:08.707 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2023-12-12T02:25:08.707 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:25:12.353 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2023-12-12T02:25:14.809 INFO:tasks.ceph.cluster1.osd.0:Stopped 2023-12-12T02:25:14.809 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2023-12-12T02:25:14.809 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:25:14.893 INFO:tasks.ceph.cluster1.osd.1:Stopped 2023-12-12T02:25:14.893 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2023-12-12T02:25:14.893 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:25:17.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2023-12-12T02:25:17.556 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2023-12-12T02:25:20.997 INFO:tasks.ceph.cluster1.osd.2:Stopped 2023-12-12T02:25:20.997 INFO:teuthology.misc:Shutting down mgr daemons... 2023-12-12T02:25:20.998 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2023-12-12T02:25:20.998 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:25:21.029 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2023-12-12T02:25:21.029 INFO:teuthology.misc:Shutting down mon daemons... 2023-12-12T02:25:21.029 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2023-12-12T02:25:21.029 INFO:teuthology.orchestra.run:waiting for 300 2023-12-12T02:25:21.094 INFO:tasks.ceph.cluster1.mon.a:Stopped 2023-12-12T02:25:21.095 INFO:tasks.ceph:Checking cluster log for badness... 2023-12-12T02:25:21.095 DEBUG:teuthology.orchestra.run.smithi055:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2023-12-12T02:25:21.152 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T02:25:21.152 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2023-12-12T02:25:21.491 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T02:25:21.492 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2023-12-12T02:25:21.708 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T02:25:21.708 DEBUG:teuthology.orchestra.run.smithi055:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2023-12-12T02:25:21.864 INFO:tasks.ceph:Archiving mon data... 2023-12-12T02:25:21.864 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/data/cluster1.mon.a.tgz 2023-12-12T02:25:21.865 DEBUG:teuthology.orchestra.run.smithi055:> mktemp 2023-12-12T02:25:21.870 INFO:teuthology.orchestra.run.smithi055.stdout:/tmp/tmp.PM7O7gITRI 2023-12-12T02:25:21.870 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.PM7O7gITRI 2023-12-12T02:25:22.000 DEBUG:teuthology.orchestra.run.smithi055:> sudo chmod 0666 /tmp/tmp.PM7O7gITRI 2023-12-12T02:25:22.063 DEBUG:teuthology.orchestra.remote:smithi055:/tmp/tmp.PM7O7gITRI is 304KB 2023-12-12T02:25:22.121 DEBUG:teuthology.orchestra.run.smithi055:> rm -fr /tmp/tmp.PM7O7gITRI 2023-12-12T02:25:22.127 INFO:tasks.ceph:Cleaning ceph cluster... 2023-12-12T02:25:22.128 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-12T02:25:22.172 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2023-12-12T02:25:22.232 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-12T02:25:22.237 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2023-12-12T02:25:22.237 INFO:tasks.ceph:Archiving crash dumps... 2023-12-12T02:25:22.239 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi055/crash 2023-12-12T02:25:22.240 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-12T02:25:22.288 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/lib/ceph/crash to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi140/crash 2023-12-12T02:25:22.289 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-12-12T02:25:22.350 INFO:tasks.ceph:Compressing logs... 2023-12-12T02:25:22.350 DEBUG:teuthology.orchestra.run.smithi055:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-12T02:25:22.353 DEBUG:teuthology.orchestra.run.smithi140:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-12T02:25:22.373 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39088.log 2023-12-12T02:25:22.374 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40672.log 2023-12-12T02:25:22.374 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38843.log 2023-12-12T02:25:22.375 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39088.log: /var/log/ceph/cluster1-client.admin.40672.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40672.log.gz 2023-12-12T02:25:22.375 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39088.log.gz 2023-12-12T02:25:22.375 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2023-12-12T02:25:22.376 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2023-12-12T02:25:22.376 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.38843.log: /var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34725.log 2023-12-12T02:25:22.376 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38843.log.gz 2023-12-12T02:25:22.376 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40594.log 2023-12-12T02:25:22.377 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34725.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39239.log 2023-12-12T02:25:22.377 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34725.log.gz 2023-12-12T02:25:22.377 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40594.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40594.log.gz 2023-12-12T02:25:22.377 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40804.log 2023-12-12T02:25:22.378 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40542.log 2023-12-12T02:25:22.378 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39239.log.gz 2023-12-12T02:25:22.378 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40804.log: 0.0%gzip -- replaced with /var/log/ceph/cluster1-client.admin.40804.log.gz 2023-12-12T02:25:22.379 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.39028.log 2023-12-12T02:25:22.379 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40542.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40542.log.gz 2023-12-12T02:25:22.379 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39084.log 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.30976.log 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39028.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39028.log.gz 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39084.log.gz 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/ceph.tmp-client.admin.30976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39054.log 2023-12-12T02:25:22.380 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.30976.log.gz 2023-12-12T02:25:22.381 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38925.log 2023-12-12T02:25:22.382 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39054.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34782.log 2023-12-12T02:25:22.382 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39054.log.gz 2023-12-12T02:25:22.382 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.38925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34867.log 2023-12-12T02:25:22.382 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38925.log.gz 2023-12-12T02:25:22.382 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34782.log: gzip -5 --verbose 0.0% -- /var/log/ceph/cluster1-client.admin.39284.log -- replaced with /var/log/ceph/cluster1-client.admin.34782.log.gz 2023-12-12T02:25:22.383 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-12-12T02:25:22.383 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34867.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2023-12-12T02:25:22.383 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34867.log.gz 2023-12-12T02:25:22.383 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39284.log: 0.0%gzip -5 -- replaced with /var/log/ceph/cluster1-client.admin.39284.log.gz 2023-12-12T02:25:22.383 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.34749.log 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34749.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34749.log.gz 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39331.log 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39240.log 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-osd.2.log: /var/log/ceph/cluster1-client.admin.39331.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39331.log.gz 2023-12-12T02:25:22.384 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2023-12-12T02:25:22.386 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34607.log 2023-12-12T02:25:22.386 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39240.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39240.log.gz 2023-12-12T02:25:22.386 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40620.log 2023-12-12T02:25:22.386 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34607.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39206.log 2023-12-12T02:25:22.386 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.34607.log.gz 2023-12-12T02:25:22.387 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39392.log 2023-12-12T02:25:22.387 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40620.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40620.log.gz/var/log/ceph/cluster1-client.admin.39206.log: 2023-12-12T02:25:22.387 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40698.log 2023-12-12T02:25:22.387 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39206.log.gz 2023-12-12T02:25:22.387 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.39392.log: /var/log/ceph/cluster1-client.admin.39160.log 2023-12-12T02:25:22.388 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39392.log.gz 2023-12-12T02:25:22.388 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40698.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39002.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40698.log.gz/var/log/ceph/cluster1-client.admin.39160.log: gzip 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.40568.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39160.log.gz 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39002.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39418.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.39002.log.gz 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40780.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40568.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40568.log.gz/var/log/ceph/cluster1-client.admin.39418.log: 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39318.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39418.log.gz 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40780.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39157.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40780.log.gz 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39087.log 2023-12-12T02:25:22.391 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39318.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39318.log.gz 2023-12-12T02:25:22.392 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39157.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39157.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40751.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40464.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39087.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39087.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40751.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40751.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39314.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40464.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34701.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.39314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40724.log 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39314.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.34701.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.34701.log.gz 2023-12-12T02:25:22.395 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40646.log 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40724.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40724.log.gz -5 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.38976.log 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40646.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40646.log.gz 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40490.log 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.38976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40516.log 2023-12-12T02:25:22.397 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38976.log.gz 2023-12-12T02:25:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40490.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40490.log.gz 2023-12-12T02:25:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2023-12-12T02:25:22.398 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.40516.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39444.log 2023-12-12T02:25:22.398 INFO:teuthology.orchestra.run.smithi055.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40516.log.gz 2023-12-12T02:25:22.399 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.38950.log 2023-12-12T02:25:22.399 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.39444.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39444.log.gz 2023-12-12T02:25:22.400 INFO:teuthology.orchestra.run.smithi055.stderr: 87.9% -- replaced with /var/log/ceph/cluster1.log.gz 2023-12-12T02:25:22.401 INFO:teuthology.orchestra.run.smithi055.stderr:/var/log/ceph/cluster1-client.admin.38950.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.38950.log.gz 2023-12-12T02:25:22.508 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63459.log 2023-12-12T02:25:22.508 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59557.log 2023-12-12T02:25:22.509 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52261.log 2023-12-12T02:25:22.509 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63459.log: /var/log/ceph/cluster2-client.mirror.59557.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52705.log 2023-12-12T02:25:22.510 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59557.log.gz 2023-12-12T02:25:22.510 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61579.log 2023-12-12T02:25:22.511 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52261.log.gz 2023-12-12T02:25:22.511 INFO:teuthology.orchestra.run.smithi140.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.63459.log.gz 2023-12-12T02:25:22.511 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69639.log 2023-12-12T02:25:22.511 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52705.log: /var/log/ceph/cluster1-client.mirror.61579.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57641.log 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46455.log 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.52705.log.gz 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69639.log.gz 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57641.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57314.log 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61579.log.gz 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46455.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59797.log 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62347.log 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46455.log.gz 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59797.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57314.log.gz 2023-12-12T02:25:22.513 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76228.log 2023-12-12T02:25:22.515 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59797.log.gz 2023-12-12T02:25:22.515 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62347.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50947.log 2023-12-12T02:25:22.515 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.57641.log.gz 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62347.log.gz 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76228.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48245.log 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50947.log: gzip -5 --verbose 87.1% -- -- replaced with /var/log/ceph/cluster2-client.mirror.76228.log.gz 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr: /var/log/ceph/cluster2-client.admin.79291.log 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46261.log 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79291.log: gzip -5 --verbose -- 87.6% /var/log/ceph/cluster2-client.mirror.52779.log 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48245.log.gz 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79291.log.gz 2023-12-12T02:25:22.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46261.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70224.log 2023-12-12T02:25:22.518 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52779.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57166.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70224.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59672.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52779.log.gz 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70224.log.gz 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57166.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78485.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59672.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42626.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.57166.log.gz 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78485.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59672.log.gz 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78485.log.gz 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73092.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57663.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42626.log: /var/log/ceph/cluster1-client.mirror.73092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65079.log 2023-12-12T02:25:22.519 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73092.log.gz 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66060.log 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65079.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/cluster2-client.mirror.57663.log.gz -- 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr: /var/log/ceph/cluster2-client.mirror.67445.log 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66060.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49909.log 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67445.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61827.log 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49909.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67445.log.gz 2023-12-12T02:25:22.523 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56482.log 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49909.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65079.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46261.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66060.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55787.log 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61827.log: /var/log/ceph/cluster2-client.mirror.56482.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45214.log 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55787.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47471.log 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61827.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45214.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55787.log.gz 2023-12-12T02:25:22.524 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65978.log 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47471.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45214.log.gz 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56738.log 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56482.log.gz 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65978.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39596.log 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65978.log.gz 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56738.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70044.log 2023-12-12T02:25:22.527 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55911.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39596.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39596.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70044.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60064.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70044.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42626.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55911.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57854.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68413.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60064.log: /var/log/ceph/cluster1-client.mirror.57854.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56738.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55911.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63769.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60064.log.gz 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65743.log 2023-12-12T02:25:22.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68413.log: /var/log/ceph/cluster1-client.mirror.63769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55851.log 2023-12-12T02:25:22.533 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63769.log.gz 2023-12-12T02:25:22.533 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57854.log.gz 2023-12-12T02:25:22.533 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65743.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71011.log 2023-12-12T02:25:22.533 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55851.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71056.log 2023-12-12T02:25:22.533 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65743.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71281.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71011.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68413.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71056.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70606.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71281.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71056.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69774.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71281.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47471.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70606.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50581.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.55851.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70606.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56524.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69774.log.gz 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49652.log 2023-12-12T02:25:22.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56524.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.73986.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49652.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50581.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49652.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76896.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56524.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44638.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.3.73986.log: /var/log/ceph/cluster2-client.mirror.76896.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71326.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr: 82.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76896.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76602.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71326.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44638.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71326.log.gz 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58551.log 2023-12-12T02:25:22.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71512.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76602.log: /var/log/ceph/cluster1-client.mirror.58551.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80565.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76602.log.gz 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.71512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60556.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80565.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67652.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80565.log.gz 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48265.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58551.log.gz 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67652.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54814.log 2023-12-12T02:25:22.539 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60556.log.gz 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.64160.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67652.log.gz 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54814.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48265.log.gzgzip 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53826.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54814.log.gz 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.2.64160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60939.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47385.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53826.log: /var/log/ceph/cluster2-client.mirror.60939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52340.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47385.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49968.log 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52340.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52340.log.gz 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60939.log.gz -5 2023-12-12T02:25:22.540 INFO:teuthology.orchestra.run.smithi140.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.73508.log 2023-12-12T02:25:22.541 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53826.log.gz 2023-12-12T02:25:22.543 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49968.log: 91.7% -- replaced with /var/log/ceph/cluster1-client.mirror.3.73986.log.gz 2023-12-12T02:25:22.544 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49030.log 2023-12-12T02:25:22.544 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73508.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69729.log 2023-12-12T02:25:22.544 INFO:teuthology.orchestra.run.smithi140.stderr: 83.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73508.log.gz 2023-12-12T02:25:22.544 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47385.log.gz 2023-12-12T02:25:22.545 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49030.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69279.log 2023-12-12T02:25:22.546 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49030.log.gz 2023-12-12T02:25:22.546 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69729.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63746.log 2023-12-12T02:25:22.546 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69729.log.gz 2023-12-12T02:25:22.546 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68248.log 2023-12-12T02:25:22.546 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69279.log.gz 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63746.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62560.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68248.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70764.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63746.log.gz 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62560.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56806.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68248.log.gz 87.7% 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62560.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.52421.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70764.log: /var/log/ceph/cluster2-client.mirror.56806.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75055.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70764.log.gz 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.2.52421.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76435.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75055.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68496.log 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56806.log.gz 2023-12-12T02:25:22.550 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76435.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.75055.log.gz 2023-12-12T02:25:22.551 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73922.log 2023-12-12T02:25:22.551 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.76435.log.gz 2023-12-12T02:25:22.551 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68496.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55007.log 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64481.log 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68496.log.gz 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55007.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73922.log.gz 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55007.log.gz 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51071.log 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64481.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60642.log 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.51071.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64481.log.gz 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72854.log 2023-12-12T02:25:22.553 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57942.log 2023-12-12T02:25:22.554 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60642.log: /var/log/ceph/cluster2-client.mirror.72854.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.60642.log.gz 2023-12-12T02:25:22.554 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57271.log 2023-12-12T02:25:22.555 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57942.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72854.log.gz 2023-12-12T02:25:22.555 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57942.log.gz 2023-12-12T02:25:22.555 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55718.log 2023-12-12T02:25:22.556 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57271.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71394.log 2023-12-12T02:25:22.556 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55718.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57271.log.gz 2023-12-12T02:25:22.556 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55718.log.gz 2023-12-12T02:25:22.557 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2023-12-12T02:25:22.558 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73799.log 2023-12-12T02:25:22.558 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71394.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76643.log 2023-12-12T02:25:22.558 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71394.log.gz 2023-12-12T02:25:22.558 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73799.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73799.log.gz 2023-12-12T02:25:22.559 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48225.log 2023-12-12T02:25:22.559 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76643.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74175.log 2023-12-12T02:25:22.559 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.76643.log.gz 2023-12-12T02:25:22.560 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48225.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48225.log.gz 2023-12-12T02:25:22.560 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47058.log 2023-12-12T02:25:22.561 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74175.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56674.log 2023-12-12T02:25:22.570 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47058.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.52421.log.gz 2023-12-12T02:25:22.570 INFO:teuthology.orchestra.run.smithi140.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47058.log.gz 2023-12-12T02:25:22.570 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50236.log 2023-12-12T02:25:22.571 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54938.log 2023-12-12T02:25:22.571 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.50236.log: -- replaced with /var/log/ceph/cluster1-client.mirror.56674.log.gz 2023-12-12T02:25:22.571 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50236.log.gz 2023-12-12T02:25:22.572 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46125.log 2023-12-12T02:25:22.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62978.log 2023-12-12T02:25:22.573 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54938.log.gz/var/log/ceph/cluster2-client.mirror.46125.log: 2023-12-12T02:25:22.573 INFO:teuthology.orchestra.run.smithi140.stderr: 89.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43982.log 2023-12-12T02:25:22.573 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46125.log.gz 2023-12-12T02:25:22.574 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62978.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70899.log 2023-12-12T02:25:22.574 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43982.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62978.log.gz 2023-12-12T02:25:22.574 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76725.log 2023-12-12T02:25:22.575 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70899.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43982.log.gz 2023-12-12T02:25:22.575 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70899.log.gzgzip 2023-12-12T02:25:22.575 INFO:teuthology.orchestra.run.smithi140.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46368.log 2023-12-12T02:25:22.576 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44546.log 2023-12-12T02:25:22.577 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76725.log: /var/log/ceph/cluster2-client.mirror.46368.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76725.log.gz 2023-12-12T02:25:22.577 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70696.log 2023-12-12T02:25:22.578 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44546.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44546.log.gz 2023-12-12T02:25:22.579 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39514.log 2023-12-12T02:25:22.580 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70696.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70696.log.gz 2023-12-12T02:25:22.580 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76187.log 2023-12-12T02:25:22.581 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39514.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39514.log.gz 2023-12-12T02:25:22.581 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54745.log 2023-12-12T02:25:22.582 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76187.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76187.log.gz 2023-12-12T02:25:22.582 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72599.log 2023-12-12T02:25:22.583 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54745.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54745.log.gz 2023-12-12T02:25:22.584 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58817.log 2023-12-12T02:25:22.585 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72599.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72599.log.gz 2023-12-12T02:25:22.585 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65151.log 2023-12-12T02:25:22.586 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58817.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58817.log.gz 2023-12-12T02:25:22.587 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44004.log 2023-12-12T02:25:22.588 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65151.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65151.log.gz 2023-12-12T02:25:22.588 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58318.log 2023-12-12T02:25:22.589 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44004.log: 86.1% -- replaced with /var/log/ceph/cluster2-client.mirror.44004.log.gz 2023-12-12T02:25:22.589 INFO:teuthology.orchestra.run.smithi140.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.64160.log.gz 2023-12-12T02:25:22.590 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68995.log 2023-12-12T02:25:22.591 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64923.log 2023-12-12T02:25:22.592 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68995.log: 87.6% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58318.log.gz 2023-12-12T02:25:22.592 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68995.log.gz 2023-12-12T02:25:22.593 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.67910.log 2023-12-12T02:25:22.593 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64923.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70966.log 2023-12-12T02:25:22.593 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.67910.log: -- replaced with /var/log/ceph/cluster2-client.mirror.64923.log.gz 2023-12-12T02:25:22.593 INFO:teuthology.orchestra.run.smithi140.stderr: 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.67910.log.gz 2023-12-12T02:25:22.594 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69954.log 2023-12-12T02:25:22.595 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66928.log 2023-12-12T02:25:22.595 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70966.log.gz 2023-12-12T02:25:22.595 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69954.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69954.log.gz 2023-12-12T02:25:22.595 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72517.log 2023-12-12T02:25:22.596 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48714.log 2023-12-12T02:25:22.596 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66928.log: /var/log/ceph/cluster2-client.mirror.72517.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72517.log.gz 2023-12-12T02:25:22.597 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66928.log.gz 2023-12-12T02:25:22.597 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66081.log 2023-12-12T02:25:22.598 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66432.log 2023-12-12T02:25:22.599 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48714.log: /var/log/ceph/cluster2-client.mirror.66081.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66081.log.gz 2023-12-12T02:25:22.599 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48714.log.gz 2023-12-12T02:25:22.599 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59998.log 2023-12-12T02:25:22.600 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55873.log 2023-12-12T02:25:22.601 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66432.log: /var/log/ceph/cluster2-client.mirror.59998.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59998.log.gz 2023-12-12T02:25:22.601 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66432.log.gz 2023-12-12T02:25:22.601 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76064.log 2023-12-12T02:25:22.602 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77752.log 2023-12-12T02:25:22.603 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55873.log: /var/log/ceph/cluster2-client.mirror.76064.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55873.log.gz 2023-12-12T02:25:22.603 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76064.log.gz 2023-12-12T02:25:22.603 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58867.log 2023-12-12T02:25:22.603 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58731.log 2023-12-12T02:25:22.604 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77752.log: /var/log/ceph/cluster2-client.mirror.58867.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77752.log.gz 2023-12-12T02:25:22.605 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45237.log 2023-12-12T02:25:22.605 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58867.log.gz 2023-12-12T02:25:22.605 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58731.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58731.log.gz 2023-12-12T02:25:22.605 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54431.log 2023-12-12T02:25:22.606 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77579.log 2023-12-12T02:25:22.606 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45237.log: /var/log/ceph/cluster2-client.mirror.54431.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45237.log.gz 2023-12-12T02:25:22.607 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76826.log 2023-12-12T02:25:22.607 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54431.log.gz 2023-12-12T02:25:22.607 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77579.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77579.log.gz 2023-12-12T02:25:22.608 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43236.log 2023-12-12T02:25:22.608 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40651.log 2023-12-12T02:25:22.609 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76826.log: /var/log/ceph/cluster1-client.mirror.43236.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76826.log.gz 2023-12-12T02:25:22.609 INFO:teuthology.orchestra.run.smithi140.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43236.log.gz 2023-12-12T02:25:22.609 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73861.log 2023-12-12T02:25:22.610 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58338.log 2023-12-12T02:25:22.610 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.40651.log: /var/log/ceph/cluster2-client.mirror.73861.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.40651.log.gz 2023-12-12T02:25:22.611 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73861.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66556.log 2023-12-12T02:25:22.611 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:22.612 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58338.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58338.log.gz 2023-12-12T02:25:22.613 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50071.log 2023-12-12T02:25:22.614 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78979.log 2023-12-12T02:25:22.614 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66556.log: /var/log/ceph/cluster2-client.mirror.50071.log: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50071.log.gz 2023-12-12T02:25:22.617 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66556.log.gz 2023-12-12T02:25:22.617 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57446.log 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73649.log 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78979.log: /var/log/ceph/cluster2-client.mirror.57446.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78979.log.gz 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47557.log 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57446.log.gz 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73649.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73649.log.gz 2023-12-12T02:25:22.618 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53894.log 2023-12-12T02:25:22.619 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55201.log 2023-12-12T02:25:22.619 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47557.log: /var/log/ceph/cluster2-client.mirror.53894.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53894.log.gz 2023-12-12T02:25:22.623 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47557.log.gz 2023-12-12T02:25:22.623 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51049.log 2023-12-12T02:25:22.624 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61310.log 2023-12-12T02:25:22.625 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55201.log: /var/log/ceph/cluster2-client.mirror.51049.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55201.log.gz 2023-12-12T02:25:22.625 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51049.log.gz 2023-12-12T02:25:22.625 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75420.log 2023-12-12T02:25:22.625 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59215.log 2023-12-12T02:25:22.626 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61310.log: /var/log/ceph/cluster1-client.mirror.75420.log: 86.7% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61310.log.gz 2023-12-12T02:25:22.626 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.75420.log.gz 2023-12-12T02:25:22.626 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77297.log 2023-12-12T02:25:22.627 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48697.log 2023-12-12T02:25:22.627 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59215.log: /var/log/ceph/cluster2-client.mirror.77297.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77297.log.gz 2023-12-12T02:25:22.627 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59215.log.gz 2023-12-12T02:25:22.628 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64553.log 2023-12-12T02:25:22.628 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63657.log 2023-12-12T02:25:22.628 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48697.log: /var/log/ceph/cluster1-client.mirror.64553.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48697.log.gz 2023-12-12T02:25:22.629 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43257.log 2023-12-12T02:25:22.629 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63657.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63657.log.gz 2023-12-12T02:25:22.630 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72813.log 2023-12-12T02:25:22.630 INFO:teuthology.orchestra.run.smithi140.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64553.log.gz 2023-12-12T02:25:22.635 INFO:teuthology.orchestra.run.smithi055.stderr: 91.3% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2023-12-12T02:25:22.636 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43257.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43257.log.gz 2023-12-12T02:25:22.636 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68558.log 2023-12-12T02:25:22.637 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42516.log 2023-12-12T02:25:22.638 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72813.log: /var/log/ceph/cluster2-client.mirror.68558.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72813.log.gz 2023-12-12T02:25:22.638 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68558.log.gz 2023-12-12T02:25:22.638 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54218.log 2023-12-12T02:25:22.639 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52999.log 2023-12-12T02:25:22.639 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42516.log: /var/log/ceph/cluster2-client.mirror.54218.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42516.log.gz 2023-12-12T02:25:22.639 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54218.log.gz 2023-12-12T02:25:22.639 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61916.log 2023-12-12T02:25:22.640 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77600.log 2023-12-12T02:25:22.640 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52999.log: /var/log/ceph/cluster1-client.mirror.61916.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52999.log.gz 2023-12-12T02:25:22.640 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61916.log.gz 2023-12-12T02:25:22.640 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55954.log 2023-12-12T02:25:22.641 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77600.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77600.log.gz 2023-12-12T02:25:22.641 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45168.log 2023-12-12T02:25:22.642 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48455.log 2023-12-12T02:25:22.642 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55954.log: /var/log/ceph/cluster1-client.mirror.45168.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55954.log.gz 2023-12-12T02:25:22.643 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45168.log.gz 2023-12-12T02:25:22.643 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75489.log 2023-12-12T02:25:22.643 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57834.log 2023-12-12T02:25:22.643 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48455.log: /var/log/ceph/cluster1-client.mirror.75489.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48455.log.gz 2023-12-12T02:25:22.646 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75489.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46930.log 2023-12-12T02:25:22.646 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:22.646 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57834.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57834.log.gz 2023-12-12T02:25:22.646 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51029.log 2023-12-12T02:25:22.647 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76783.log 2023-12-12T02:25:22.647 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46930.log: /var/log/ceph/cluster2-client.mirror.51029.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51029.log.gz 2023-12-12T02:25:22.652 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46930.log.gz 2023-12-12T02:25:22.652 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67672.log 2023-12-12T02:25:22.652 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62583.log 2023-12-12T02:25:22.653 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67672.log: /var/log/ceph/cluster2-client.mirror.76783.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76783.log.gz 2023-12-12T02:25:22.654 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53914.log 2023-12-12T02:25:22.655 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62583.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62583.log.gz 2023-12-12T02:25:22.655 INFO:teuthology.orchestra.run.smithi140.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67672.log.gz 2023-12-12T02:25:22.655 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40608.log 2023-12-12T02:25:22.656 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76623.log 2023-12-12T02:25:22.656 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53914.log: /var/log/ceph/cluster2-client.admin.40608.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40608.log.gz 2023-12-12T02:25:22.656 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53914.log.gz 2023-12-12T02:25:22.656 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53693.log 2023-12-12T02:25:22.656 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44839.log 2023-12-12T02:25:22.657 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76623.log: /var/log/ceph/cluster2-client.mirror.53693.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76623.log.gz 2023-12-12T02:25:22.657 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53693.log.gz 2023-12-12T02:25:22.658 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69301.log 2023-12-12T02:25:22.659 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67548.log 2023-12-12T02:25:22.659 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.44839.log: /var/log/ceph/cluster1-client.mirror.69301.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69301.log.gz 2023-12-12T02:25:22.659 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44839.log.gz 2023-12-12T02:25:22.659 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70246.log 2023-12-12T02:25:22.660 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53591.log 2023-12-12T02:25:22.660 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67548.log: /var/log/ceph/cluster1-client.mirror.70246.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70246.log.gz 2023-12-12T02:25:22.661 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67548.log.gz 2023-12-12T02:25:22.662 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79317.log 2023-12-12T02:25:22.662 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57767.log 2023-12-12T02:25:22.662 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53591.log: /var/log/ceph/cluster2-client.admin.79317.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53591.log.gz 2023-12-12T02:25:22.662 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79317.log.gz 2023-12-12T02:25:22.663 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72875.log 2023-12-12T02:25:22.664 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71169.log 2023-12-12T02:25:22.664 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57767.log: /var/log/ceph/cluster2-client.mirror.72875.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.72875.log.gz 2023-12-12T02:25:22.664 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.57767.log.gz 2023-12-12T02:25:22.664 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51979.log 2023-12-12T02:25:22.665 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77030.log 2023-12-12T02:25:22.665 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71169.log: /var/log/ceph/cluster2-client.mirror.51979.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71169.log.gz 2023-12-12T02:25:22.665 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61691.log 2023-12-12T02:25:22.666 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77030.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77030.log.gz 2023-12-12T02:25:22.666 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59537.log 2023-12-12T02:25:22.667 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61691.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61691.log.gz 2023-12-12T02:25:22.668 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.77859.log 2023-12-12T02:25:22.669 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59537.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59537.log.gz 2023-12-12T02:25:22.669 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75610.log 2023-12-12T02:25:22.671 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.77859.log: 91.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.77859.log.gz 2023-12-12T02:25:22.672 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51621.log 2023-12-12T02:25:22.673 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75610.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75610.log.gz 2023-12-12T02:25:22.673 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54287.log 2023-12-12T02:25:22.674 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51621.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51621.log.gz 2023-12-12T02:25:22.675 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40565.log 2023-12-12T02:25:22.676 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54287.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54287.log.gz 2023-12-12T02:25:22.676 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60806.log 2023-12-12T02:25:22.677 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40565.log: 56.3% -- replaced with /var/log/ceph/cluster1-client.mirror.40565.log.gz 2023-12-12T02:25:22.677 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42793.log 2023-12-12T02:25:22.691 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60806.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58034.log 2023-12-12T02:25:22.694 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.42793.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42793.log.gz 2023-12-12T02:25:22.699 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66949.log 2023-12-12T02:25:22.700 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58034.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58034.log.gz 2023-12-12T02:25:22.710 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60806.log.gz 2023-12-12T02:25:22.711 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63215.log 2023-12-12T02:25:22.712 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66949.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66949.log.gz 2023-12-12T02:25:22.712 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75078.log 2023-12-12T02:25:22.713 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63215.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63215.log.gz 2023-12-12T02:25:22.713 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46281.log 2023-12-12T02:25:22.714 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75078.log: 82.4% -- replaced with /var/log/ceph/cluster1-client.mirror.75078.log.gz 2023-12-12T02:25:22.715 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49733.log 2023-12-12T02:25:22.721 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46281.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46281.log.gz 2023-12-12T02:25:22.721 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70584.log 2023-12-12T02:25:22.735 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46571.log 2023-12-12T02:25:22.736 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70584.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.70584.log.gz 2023-12-12T02:25:22.741 INFO:teuthology.orchestra.run.smithi055.stderr: 91.3% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2023-12-12T02:25:22.743 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74175.log.gz 2023-12-12T02:25:22.743 INFO:teuthology.orchestra.run.smithi140.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.49733.log.gz 2023-12-12T02:25:22.744 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.45006.log 2023-12-12T02:25:22.744 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61554.log 2023-12-12T02:25:22.745 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46571.log: /var/log/ceph/cluster1-client.mirror.1.45006.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46571.log.gz 2023-12-12T02:25:22.745 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49002.log 2023-12-12T02:25:22.746 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61554.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61554.log.gz 2023-12-12T02:25:22.746 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76849.log 2023-12-12T02:25:22.752 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49002.log: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.45006.log.gz 2023-12-12T02:25:22.755 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.43670.log 2023-12-12T02:25:22.756 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76849.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76849.log.gz 2023-12-12T02:25:22.756 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39825.log 2023-12-12T02:25:22.769 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.43670.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49968.log.gz 2023-12-12T02:25:22.771 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45481.log 2023-12-12T02:25:22.771 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.39825.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39825.log.gz 2023-12-12T02:25:22.772 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48756.log 2023-12-12T02:25:22.772 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.1.43670.log.gz 2023-12-12T02:25:22.773 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45481.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45481.log.gz 2023-12-12T02:25:22.773 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.67722.log 2023-12-12T02:25:22.773 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70336.log 2023-12-12T02:25:22.787 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.67722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68393.log 2023-12-12T02:25:22.787 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70336.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70336.log.gz 2023-12-12T02:25:22.803 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56939.log 2023-12-12T02:25:22.803 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68393.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68393.log.gz 2023-12-12T02:25:22.804 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47815.log 2023-12-12T02:25:22.806 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56939.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56939.log.gz 2023-12-12T02:25:22.806 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73779.log 2023-12-12T02:25:22.812 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47815.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47815.log.gz 2023-12-12T02:25:22.819 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.67722.log.gz 2023-12-12T02:25:22.819 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49733.log 2023-12-12T02:25:22.820 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73779.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73779.log.gz 2023-12-12T02:25:22.820 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69976.log 2023-12-12T02:25:22.835 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.49733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50475.log 2023-12-12T02:25:22.835 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69976.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69976.log.gz 2023-12-12T02:25:22.851 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74973.log 2023-12-12T02:25:22.863 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50475.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71512.log.gz 2023-12-12T02:25:22.867 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59236.log 2023-12-12T02:25:22.868 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74973.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.50475.log.gz 2023-12-12T02:25:22.868 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74973.log.gz 2023-12-12T02:25:22.868 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73881.log 2023-12-12T02:25:22.870 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59236.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59236.log.gz 2023-12-12T02:25:22.870 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72661.log 2023-12-12T02:25:22.871 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73881.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73881.log.gz 2023-12-12T02:25:22.872 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60600.log 2023-12-12T02:25:22.872 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72661.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72661.log.gz 2023-12-12T02:25:22.873 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60266.log 2023-12-12T02:25:22.875 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60600.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60600.log.gz 2023-12-12T02:25:22.876 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53716.log 2023-12-12T02:25:22.876 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60266.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60266.log.gz 2023-12-12T02:25:22.877 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55158.log 2023-12-12T02:25:22.877 INFO:teuthology.orchestra.run.smithi140.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49733.log.gz 2023-12-12T02:25:22.877 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53716.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53716.log.gz 2023-12-12T02:25:22.878 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58011.log 2023-12-12T02:25:22.878 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52626.log 2023-12-12T02:25:22.879 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55158.log: /var/log/ceph/cluster1-client.mirror.58011.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.58011.log.gz 2023-12-12T02:25:22.879 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55158.log.gz 2023-12-12T02:25:22.880 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50846.log 2023-12-12T02:25:22.880 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65263.log 2023-12-12T02:25:22.881 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52626.log: /var/log/ceph/cluster2-client.mirror.50846.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52626.log.gz 2023-12-12T02:25:22.881 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50846.log.gz 2023-12-12T02:25:22.881 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52937.log 2023-12-12T02:25:22.881 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78927.log 2023-12-12T02:25:22.882 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65263.log: /var/log/ceph/cluster2-client.mirror.52937.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65263.log.gz 2023-12-12T02:25:22.882 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79109.log 2023-12-12T02:25:22.882 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52937.log.gz 2023-12-12T02:25:22.882 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78927.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78927.log.gz 2023-12-12T02:25:22.883 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68351.log 2023-12-12T02:25:22.883 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43824.log 2023-12-12T02:25:22.884 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79109.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79109.log.gz 2023-12-12T02:25:22.884 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68351.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70471.log 2023-12-12T02:25:22.885 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43824.log: 88.6% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.43824.log.gz 2023-12-12T02:25:22.885 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68351.log.gz 2023-12-12T02:25:22.885 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47451.log 2023-12-12T02:25:22.886 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56175.log 2023-12-12T02:25:22.886 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70471.log: /var/log/ceph/cluster2-client.mirror.47451.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70471.log.gz 2023-12-12T02:25:22.886 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78407.log 2023-12-12T02:25:22.887 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56175.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56175.log.gz 2023-12-12T02:25:22.887 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59340.log 2023-12-12T02:25:22.888 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78407.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78407.log.gz 2023-12-12T02:25:22.889 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70989.log 2023-12-12T02:25:22.889 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59340.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59340.log.gz 2023-12-12T02:25:22.889 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68661.log 2023-12-12T02:25:22.890 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70989.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70989.log.gz 2023-12-12T02:25:22.890 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70674.log 2023-12-12T02:25:22.891 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47451.log.gz 2023-12-12T02:25:22.892 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68661.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68661.log.gz 2023-12-12T02:25:22.893 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57790.log 2023-12-12T02:25:22.894 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61442.log 2023-12-12T02:25:22.894 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70674.log: /var/log/ceph/cluster1-client.mirror.57790.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70674.log.gz 2023-12-12T02:25:22.894 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57790.log.gz 2023-12-12T02:25:22.894 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56086.log 2023-12-12T02:25:22.895 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55675.log 2023-12-12T02:25:22.895 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61442.log: /var/log/ceph/cluster2-client.mirror.56086.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61442.log.gz 2023-12-12T02:25:22.895 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56086.log.gz 2023-12-12T02:25:22.896 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61166.log 2023-12-12T02:25:22.896 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78121.log 2023-12-12T02:25:22.897 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55675.log: /var/log/ceph/cluster1-client.mirror.61166.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61166.log.gz 2023-12-12T02:25:22.897 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55675.log.gz 2023-12-12T02:25:22.897 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53635.log 2023-12-12T02:25:22.898 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47666.log 2023-12-12T02:25:22.898 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.78121.log: /var/log/ceph/cluster2-client.mirror.53635.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.78121.log.gz 2023-12-12T02:25:22.898 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55338.log 2023-12-12T02:25:22.898 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53635.log.gz 2023-12-12T02:25:22.903 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47666.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47666.log.gz 2023-12-12T02:25:22.904 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50295.log 2023-12-12T02:25:22.904 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77620.log 2023-12-12T02:25:22.905 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55338.log: /var/log/ceph/cluster2-client.mirror.50295.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55338.log.gz 2023-12-12T02:25:22.905 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52900.log 2023-12-12T02:25:22.906 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77620.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77620.log.gz 2023-12-12T02:25:22.906 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65593.log 2023-12-12T02:25:22.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52900.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69234.log 2023-12-12T02:25:22.920 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50947.log.gz 2023-12-12T02:25:22.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65593.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65593.log.gz 2023-12-12T02:25:22.923 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43627.log 2023-12-12T02:25:22.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69234.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69234.log.gz 2023-12-12T02:25:22.924 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63114.log 2023-12-12T02:25:22.925 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43627.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43627.log.gz 2023-12-12T02:25:22.925 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48205.log 2023-12-12T02:25:22.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63114.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63114.log.gz 2023-12-12T02:25:22.926 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60041.log 2023-12-12T02:25:22.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48205.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48205.log.gz 2023-12-12T02:25:22.927 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69909.log 2023-12-12T02:25:22.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60041.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60041.log.gz 2023-12-12T02:25:22.929 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51482.log 2023-12-12T02:25:22.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69909.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69909.log.gz 2023-12-12T02:25:22.930 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50747.log 2023-12-12T02:25:22.930 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48756.log.gz 2023-12-12T02:25:22.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49089.log 2023-12-12T02:25:22.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50747.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51482.log.gz 2023-12-12T02:25:22.931 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50826.log 2023-12-12T02:25:22.932 INFO:teuthology.orchestra.run.smithi140.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.50747.log.gz 2023-12-12T02:25:22.932 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49089.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49573.log 2023-12-12T02:25:22.932 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51979.log.gz 2023-12-12T02:25:22.933 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50826.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50826.log.gz 2023-12-12T02:25:22.933 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53848.log 2023-12-12T02:25:22.934 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49573.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56697.log 2023-12-12T02:25:22.935 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53848.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49573.log.gz 2023-12-12T02:25:22.935 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.53848.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73943.log 2023-12-12T02:25:22.935 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:22.935 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56697.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56697.log.gz 2023-12-12T02:25:22.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45773.log 2023-12-12T02:25:22.936 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58251.log 2023-12-12T02:25:22.936 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73943.log: /var/log/ceph/cluster2-client.mirror.45773.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73943.log.gz 2023-12-12T02:25:22.937 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57424.log 2023-12-12T02:25:22.937 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45773.log.gz 2023-12-12T02:25:22.937 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58251.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58251.log.gz 2023-12-12T02:25:22.938 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51900.log 2023-12-12T02:25:22.938 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57813.log 2023-12-12T02:25:22.940 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57424.log: /var/log/ceph/cluster2-client.mirror.51900.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.57424.log.gz 2023-12-12T02:25:22.941 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51071.log.gz 2023-12-12T02:25:22.941 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80253.log 2023-12-12T02:25:22.941 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57813.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.57813.log.gz 2023-12-12T02:25:22.942 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48144.log 2023-12-12T02:25:22.943 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68434.log 2023-12-12T02:25:22.943 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80253.log: /var/log/ceph/cluster2-client.mirror.48144.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80253.log.gz 2023-12-12T02:25:22.943 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65349.log 2023-12-12T02:25:22.943 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48144.log.gz 2023-12-12T02:25:22.944 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68434.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68434.log.gz 2023-12-12T02:25:22.944 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60487.log 2023-12-12T02:25:22.944 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54451.log 2023-12-12T02:25:22.945 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65349.log: /var/log/ceph/cluster2-client.mirror.60487.log: 87.8% 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65349.log.gz 2023-12-12T02:25:22.945 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60487.log.gz 2023-12-12T02:25:22.945 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40088.log 2023-12-12T02:25:22.946 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77948.log 2023-12-12T02:25:22.946 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54451.log: /var/log/ceph/cluster2-client.mirror.40088.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54451.log.gz 2023-12-12T02:25:22.946 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42493.log 2023-12-12T02:25:22.947 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77948.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77948.log.gz 2023-12-12T02:25:22.947 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46594.log 2023-12-12T02:25:22.948 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42493.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42493.log.gz 2023-12-12T02:25:22.948 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43535.log 2023-12-12T02:25:22.949 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46594.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.46594.log.gz 2023-12-12T02:25:22.950 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57074.log 2023-12-12T02:25:22.950 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43535.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.43535.log.gz 2023-12-12T02:25:22.950 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57207.log 2023-12-12T02:25:22.951 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57074.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57074.log.gz 2023-12-12T02:25:22.951 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68165.log 2023-12-12T02:25:22.953 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57207.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57207.log.gz 2023-12-12T02:25:22.954 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66288.log 2023-12-12T02:25:22.955 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68165.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68165.log.gz 2023-12-12T02:25:22.956 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42470.log 2023-12-12T02:25:22.956 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66288.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66288.log.gz 2023-12-12T02:25:22.957 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58910.log 2023-12-12T02:25:22.957 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42470.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42470.log.gz 2023-12-12T02:25:22.958 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56369.log 2023-12-12T02:25:22.958 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58910.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58910.log.gz 2023-12-12T02:25:22.959 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46217.log 2023-12-12T02:25:22.960 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56369.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56369.log.gz 2023-12-12T02:25:22.960 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52762.log 2023-12-12T02:25:22.961 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46217.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46217.log.gz 2023-12-12T02:25:22.961 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46993.log 2023-12-12T02:25:22.962 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52762.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52762.log.gz 2023-12-12T02:25:22.962 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53673.log 2023-12-12T02:25:22.967 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46993.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46993.log.gz 2023-12-12T02:25:22.968 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68954.log 2023-12-12T02:25:22.969 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53673.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53673.log.gz 2023-12-12T02:25:22.969 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44279.log 2023-12-12T02:25:22.970 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68954.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68954.log.gz 2023-12-12T02:25:22.970 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34875.log 2023-12-12T02:25:22.975 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44279.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44279.log.gz 2023-12-12T02:25:22.976 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63571.log 2023-12-12T02:25:22.977 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.34875.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34875.log.gz 2023-12-12T02:25:22.977 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45344.log 2023-12-12T02:25:22.977 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63571.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63571.log.gz 2023-12-12T02:25:22.978 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51462.log 2023-12-12T02:25:22.983 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45344.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45344.log.gz 2023-12-12T02:25:22.984 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48351.log 2023-12-12T02:25:22.985 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51462.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51462.log.gz 2023-12-12T02:25:22.985 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66226.log 2023-12-12T02:25:22.985 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.48351.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48351.log.gz 2023-12-12T02:25:22.986 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59889.log 2023-12-12T02:25:22.987 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66226.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66226.log.gz 2023-12-12T02:25:22.987 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56562.log 2023-12-12T02:25:22.988 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59889.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59889.log.gz 2023-12-12T02:25:22.988 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66577.log 2023-12-12T02:25:22.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56562.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56562.log.gz 2023-12-12T02:25:22.989 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52078.log 2023-12-12T02:25:22.990 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66577.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66577.log.gz 2023-12-12T02:25:22.990 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61531.log 2023-12-12T02:25:22.991 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52078.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52078.log.gz 2023-12-12T02:25:22.992 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44869.log 2023-12-12T02:25:22.992 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61531.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61531.log.gz 2023-12-12T02:25:22.992 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66412.log 2023-12-12T02:25:22.993 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44869.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44869.log.gz 2023-12-12T02:25:22.993 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40388.log 2023-12-12T02:25:22.994 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66412.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66412.log.gz 2023-12-12T02:25:22.994 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53136.log 2023-12-12T02:25:22.995 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.40388.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40388.log.gz 2023-12-12T02:25:22.995 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64766.log 2023-12-12T02:25:22.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53136.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53136.log.gz 2023-12-12T02:25:22.996 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62025.log 2023-12-12T02:25:22.998 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64766.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64766.log.gz 2023-12-12T02:25:22.998 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76002.log 2023-12-12T02:25:22.999 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62025.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62025.log.gz 2023-12-12T02:25:23.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57508.log 2023-12-12T02:25:23.000 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76002.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76002.log.gz 2023-12-12T02:25:23.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70719.log 2023-12-12T02:25:23.001 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57508.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57508.log.gz 2023-12-12T02:25:23.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79031.log 2023-12-12T02:25:23.002 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70719.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70719.log.gz 2023-12-12T02:25:23.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73439.log 2023-12-12T02:25:23.003 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79031.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79031.log.gz 2023-12-12T02:25:23.004 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77643.log 2023-12-12T02:25:23.019 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63479.log 2023-12-12T02:25:23.020 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77643.log: 73.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77643.log.gz 2023-12-12T02:25:23.020 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51683.log 2023-12-12T02:25:23.020 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63479.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63479.log.gz 2023-12-12T02:25:23.023 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46368.log.gz 2023-12-12T02:25:23.035 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47709.log 2023-12-12T02:25:23.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51683.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51683.log.gz 2023-12-12T02:25:23.036 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77795.log 2023-12-12T02:25:23.042 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47709.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47709.log.gz 2023-12-12T02:25:23.042 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61379.log 2023-12-12T02:25:23.043 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77795.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77795.log.gz 2023-12-12T02:25:23.043 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71418.log 2023-12-12T02:25:23.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61379.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61379.log.gz 2023-12-12T02:25:23.045 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61847.log 2023-12-12T02:25:23.045 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71418.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71418.log.gz 2023-12-12T02:25:23.046 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59256.log 2023-12-12T02:25:23.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61847.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61847.log.gz 2023-12-12T02:25:23.047 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78849.log 2023-12-12T02:25:23.049 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59256.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59256.log.gz 2023-12-12T02:25:23.050 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72558.log 2023-12-12T02:25:23.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78849.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78849.log.gz 2023-12-12T02:25:23.051 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47729.log 2023-12-12T02:25:23.052 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72558.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72558.log.gz 2023-12-12T02:25:23.052 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2023-12-12T02:25:23.057 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47729.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47729.log.gz 2023-12-12T02:25:23.058 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80669.log 2023-12-12T02:25:23.071 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50108.log 2023-12-12T02:25:23.071 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80669.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80669.log.gz 2023-12-12T02:25:23.081 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51900.log.gz 2023-12-12T02:25:23.084 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49002.log.gz 2023-12-12T02:25:23.087 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74847.log 2023-12-12T02:25:23.088 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54097.log 2023-12-12T02:25:23.088 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50108.log: /var/log/ceph/cluster1-client.mirror.74847.log: 87.9% 86.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50108.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.74847.log.gz 2023-12-12T02:25:23.088 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.088 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77968.log 2023-12-12T02:25:23.088 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66350.log 2023-12-12T02:25:23.089 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77968.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54097.log.gz 2023-12-12T02:25:23.090 INFO:teuthology.orchestra.run.smithi140.stderr: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77968.log.gz 2023-12-12T02:25:23.090 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73048.log 2023-12-12T02:25:23.091 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66515.log 2023-12-12T02:25:23.091 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66350.log: /var/log/ceph/cluster2-client.mirror.73048.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66350.log.gz 2023-12-12T02:25:23.092 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73048.log.gz 2023-12-12T02:25:23.092 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66453.log 2023-12-12T02:25:23.092 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66515.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66515.log.gz 2023-12-12T02:25:23.093 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46973.log 2023-12-12T02:25:23.093 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62067.log 2023-12-12T02:25:23.093 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66453.log: /var/log/ceph/cluster2-client.mirror.46973.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66453.log.gz 2023-12-12T02:25:23.094 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49331.log 2023-12-12T02:25:23.095 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62067.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62067.log.gz 2023-12-12T02:25:23.095 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.61579.log 2023-12-12T02:25:23.096 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49331.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49331.log.gz 2023-12-12T02:25:23.096 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58162.log 2023-12-12T02:25:23.099 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.2.61579.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46973.log.gz 2023-12-12T02:25:23.111 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62087.log 2023-12-12T02:25:23.112 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58162.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58162.log.gz 2023-12-12T02:25:23.112 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41082.log 2023-12-12T02:25:23.113 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62087.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62087.log.gz 2023-12-12T02:25:23.114 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55462.log 2023-12-12T02:25:23.115 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41082.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41082.log.gz 2023-12-12T02:25:23.115 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51101.log 2023-12-12T02:25:23.116 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55462.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55462.log.gz 2023-12-12T02:25:23.116 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56582.log 2023-12-12T02:25:23.117 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51101.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51101.log.gz 2023-12-12T02:25:23.118 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67966.log 2023-12-12T02:25:23.119 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56582.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56582.log.gz 2023-12-12T02:25:23.119 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63859.log 2023-12-12T02:25:23.120 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67966.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67966.log.gz 2023-12-12T02:25:23.120 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78589.log 2023-12-12T02:25:23.122 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63859.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63859.log.gz 2023-12-12T02:25:23.122 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77356.log 2023-12-12T02:25:23.123 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78589.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78589.log.gz 2023-12-12T02:25:23.124 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74116.log 2023-12-12T02:25:23.124 INFO:teuthology.orchestra.run.smithi140.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.61579.log.gz 2023-12-12T02:25:23.136 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77356.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52900.log.gz 2023-12-12T02:25:23.137 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59023.log 2023-12-12T02:25:23.138 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39231.log 2023-12-12T02:25:23.138 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74116.log: /var/log/ceph/cluster2-client.mirror.59023.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74116.log.gz 2023-12-12T02:25:23.139 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59023.log.gz 2023-12-12T02:25:23.140 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75187.log 2023-12-12T02:25:23.140 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65015.log 2023-12-12T02:25:23.140 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39231.log.gz 2023-12-12T02:25:23.140 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75187.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75187.log.gz 2023-12-12T02:25:23.141 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50730.log 2023-12-12T02:25:23.141 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52115.log 2023-12-12T02:25:23.142 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50730.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65015.log.gz 2023-12-12T02:25:23.142 INFO:teuthology.orchestra.run.smithi140.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50730.log.gz 2023-12-12T02:25:23.142 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48492.log 2023-12-12T02:25:23.143 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55094.log 2023-12-12T02:25:23.143 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52115.log: /var/log/ceph/cluster2-client.mirror.48492.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52115.log.gz 2023-12-12T02:25:23.143 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48492.log.gz 2023-12-12T02:25:23.143 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73154.log 2023-12-12T02:25:23.144 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47322.log 2023-12-12T02:25:23.144 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55094.log: /var/log/ceph/cluster2-client.mirror.73154.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73154.log.gz 2023-12-12T02:25:23.145 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55094.log.gz 2023-12-12T02:25:23.146 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57404.log 2023-12-12T02:25:23.146 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73397.log 2023-12-12T02:25:23.148 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47322.log: /var/log/ceph/cluster2-client.mirror.57404.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57404.log.gz 2023-12-12T02:25:23.151 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47322.log.gz 2023-12-12T02:25:23.152 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59432.log 2023-12-12T02:25:23.152 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75740.log 2023-12-12T02:25:23.153 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73397.log: /var/log/ceph/cluster1-client.mirror.59432.log: 87.6% 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59432.log.gz 2023-12-12T02:25:23.153 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.73397.log.gz 2023-12-12T02:25:23.153 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40086.log 2023-12-12T02:25:23.154 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63044.log 2023-12-12T02:25:23.154 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75740.log: /var/log/ceph/cluster1-client.mirror.2.40086.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75740.log.gz 2023-12-12T02:25:23.154 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79915.log 2023-12-12T02:25:23.155 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63044.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63044.log.gz 2023-12-12T02:25:23.156 INFO:teuthology.orchestra.run.smithi140.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40088.log.gz 2023-12-12T02:25:23.156 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79655.log 2023-12-12T02:25:23.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79915.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79915.log.gz 2023-12-12T02:25:23.157 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80383.log 2023-12-12T02:25:23.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79655.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63371.log 2023-12-12T02:25:23.157 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79655.log.gz 2023-12-12T02:25:23.158 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69571.log 2023-12-12T02:25:23.158 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80383.log.gz 2023-12-12T02:25:23.158 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63371.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46746.log 2023-12-12T02:25:23.158 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69571.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69571.log.gz 2023-12-12T02:25:23.160 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59146.log 2023-12-12T02:25:23.160 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46746.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63371.log.gz 2023-12-12T02:25:23.160 INFO:teuthology.orchestra.run.smithi140.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.46746.log.gz 2023-12-12T02:25:23.160 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49373.log 2023-12-12T02:25:23.160 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59146.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47600.log 2023-12-12T02:25:23.161 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49373.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59146.log.gz 2023-12-12T02:25:23.161 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65833.log 2023-12-12T02:25:23.162 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47600.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40086.log.gz 2023-12-12T02:25:23.166 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47600.log.gz 2023-12-12T02:25:23.167 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62606.log 2023-12-12T02:25:23.167 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8%/var/log/ceph/cluster2-client.mirror.65833.log: -- replaced with /var/log/ceph/cluster2-client.mirror.50295.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2023-12-12T02:25:23.168 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.168 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62606.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65833.log.gz 2023-12-12T02:25:23.168 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62606.log.gz 2023-12-12T02:25:23.168 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70854.log 2023-12-12T02:25:23.169 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49946.log 2023-12-12T02:25:23.169 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62777.log 2023-12-12T02:25:23.169 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70854.log.gz 2023-12-12T02:25:23.170 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49946.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49946.log.gz 2023-12-12T02:25:23.170 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80825.log 2023-12-12T02:25:23.171 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61939.log 2023-12-12T02:25:23.171 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80825.log: 0.0% 87.4% -- replaced with /var/log/ceph/cluster2-client.admin.80825.log.gz 2023-12-12T02:25:23.171 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.62777.log.gz 2023-12-12T02:25:23.172 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54351.log 2023-12-12T02:25:23.172 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55764.log 2023-12-12T02:25:23.172 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61939.log: /var/log/ceph/cluster1-client.mirror.54351.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61939.log.gz 2023-12-12T02:25:23.172 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74504.log 2023-12-12T02:25:23.174 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55764.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55764.log.gz 2023-12-12T02:25:23.174 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54351.log.gz 2023-12-12T02:25:23.174 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79499.log 2023-12-12T02:25:23.175 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59626.log 2023-12-12T02:25:23.175 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74504.log: /var/log/ceph/cluster2-client.admin.79499.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79499.log.gz 2023-12-12T02:25:23.175 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74504.log.gz 2023-12-12T02:25:23.176 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50273.log 2023-12-12T02:25:23.176 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68041.log 2023-12-12T02:25:23.176 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59626.log: /var/log/ceph/cluster2-client.mirror.50273.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59626.log.gz 2023-12-12T02:25:23.176 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50273.log.gz 2023-12-12T02:25:23.176 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80643.log 2023-12-12T02:25:23.177 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59123.log 2023-12-12T02:25:23.177 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68041.log: /var/log/ceph/cluster2-client.admin.80643.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80643.log.gz 2023-12-12T02:25:23.178 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68041.log.gz 2023-12-12T02:25:23.179 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39444.log 2023-12-12T02:25:23.179 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70111.log 2023-12-12T02:25:23.179 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59123.log: /var/log/ceph/cluster2-client.admin.39444.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39444.log.gz 2023-12-12T02:25:23.180 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59123.log.gz 2023-12-12T02:25:23.180 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48104.log 2023-12-12T02:25:23.180 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68186.log 2023-12-12T02:25:23.180 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70111.log: /var/log/ceph/cluster2-client.mirror.48104.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70111.log.gz 2023-12-12T02:25:23.181 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65551.log 2023-12-12T02:25:23.181 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48104.log.gz 2023-12-12T02:25:23.182 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68186.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68186.log.gz 2023-12-12T02:25:23.182 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41026.log 2023-12-12T02:25:23.183 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75823.log 2023-12-12T02:25:23.183 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65551.log: /var/log/ceph/cluster2-client.mirror.41026.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41026.log.gz 2023-12-12T02:25:23.184 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65551.log.gz 2023-12-12T02:25:23.184 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52685.log 2023-12-12T02:25:23.186 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80695.log 2023-12-12T02:25:23.186 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75823.log: /var/log/ceph/cluster2-client.mirror.52685.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75823.log.gz 2023-12-12T02:25:23.186 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77732.log 2023-12-12T02:25:23.186 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52685.log.gz 2023-12-12T02:25:23.186 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80695.log.gz 2023-12-12T02:25:23.187 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48001.log 2023-12-12T02:25:23.187 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59043.log 2023-12-12T02:25:23.187 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48001.log: /var/log/ceph/cluster2-client.mirror.77732.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77732.log.gz 2023-12-12T02:25:23.188 INFO:teuthology.orchestra.run.smithi140.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48001.log.gz 2023-12-12T02:25:23.188 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55528.log 2023-12-12T02:25:23.189 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59043.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.59043.log.gz 2023-12-12T02:25:23.190 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65720.log 2023-12-12T02:25:23.190 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69414.log 2023-12-12T02:25:23.191 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55528.log: /var/log/ceph/cluster1-client.mirror.65720.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55528.log.gz 2023-12-12T02:25:23.191 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65720.log.gz 2023-12-12T02:25:23.191 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62437.log 2023-12-12T02:25:23.192 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45849.log 2023-12-12T02:25:23.192 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69414.log: /var/log/ceph/cluster2-client.mirror.62437.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69414.log.gz 2023-12-12T02:25:23.192 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75979.log 2023-12-12T02:25:23.193 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45849.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45849.log.gz 2023-12-12T02:25:23.193 INFO:teuthology.orchestra.run.smithi140.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.62437.log.gz 2023-12-12T02:25:23.194 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39875.log 2023-12-12T02:25:23.195 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77314.log 2023-12-12T02:25:23.195 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75979.log: /var/log/ceph/cluster1-client.admin.39875.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39875.log.gz 2023-12-12T02:25:23.195 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75979.log.gz 2023-12-12T02:25:23.195 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60087.log 2023-12-12T02:25:23.195 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.42925.log 2023-12-12T02:25:23.196 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77314.log: /var/log/ceph/cluster1-client.mirror.60087.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77314.log.gz 2023-12-12T02:25:23.197 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60087.log.gz 2023-12-12T02:25:23.197 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59192.log 2023-12-12T02:25:23.197 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47150.log 2023-12-12T02:25:23.197 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.42925.log: /var/log/ceph/cluster1-client.mirror.59192.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59192.log.gz 2023-12-12T02:25:23.198 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.42925.log.gz 2023-12-12T02:25:23.199 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55977.log 2023-12-12T02:25:23.199 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56895.log 2023-12-12T02:25:23.200 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47150.log: /var/log/ceph/cluster1-client.mirror.55977.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55977.log.gz 2023-12-12T02:25:23.204 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47150.log.gz 2023-12-12T02:25:23.205 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42225.log 2023-12-12T02:25:23.205 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73714.log 2023-12-12T02:25:23.206 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56895.log: /var/log/ceph/cluster1-client.mirror.42225.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56895.log.gz 2023-12-12T02:25:23.206 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42225.log.gz 2023-12-12T02:25:23.206 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57965.log 2023-12-12T02:25:23.207 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79577.log 2023-12-12T02:25:23.207 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73714.log: /var/log/ceph/cluster1-client.mirror.57965.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.73714.log.gz 2023-12-12T02:25:23.207 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57965.log.gz 2023-12-12T02:25:23.207 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79837.log 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77996.log 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79577.log: /var/log/ceph/cluster2-client.admin.79837.log: 0.0% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79577.log.gz 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.79837.log.gz 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80435.log 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62869.log 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77996.log: /var/log/ceph/cluster2-client.admin.80435.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.77996.log.gz 2023-12-12T02:25:23.209 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80435.log.gz 2023-12-12T02:25:23.210 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41043.log 2023-12-12T02:25:23.211 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64298.log 2023-12-12T02:25:23.211 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62869.log: /var/log/ceph/cluster2-client.mirror.41043.log: 54.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41043.log.gz 2023-12-12T02:25:23.211 INFO:teuthology.orchestra.run.smithi140.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.62869.log.gz 2023-12-12T02:25:23.211 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53612.log 2023-12-12T02:25:23.211 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68021.log 2023-12-12T02:25:23.212 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64298.log: /var/log/ceph/cluster1-client.mirror.53612.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53612.log.gz 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64298.log.gz 2023-12-12T02:25:23.212 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.212 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.75330.log 2023-12-12T02:25:23.212 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71259.log 2023-12-12T02:25:23.213 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68021.log: /var/log/ceph/cluster1-client.mirror.2.75330.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68021.log.gz 2023-12-12T02:25:23.213 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56605.log 2023-12-12T02:25:23.213 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71259.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.71259.log.gz 2023-12-12T02:25:23.214 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48960.log 2023-12-12T02:25:23.215 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56605.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56605.log.gz 2023-12-12T02:25:23.215 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66474.log 2023-12-12T02:25:23.216 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48960.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48960.log.gz 2023-12-12T02:25:23.217 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57143.log 2023-12-12T02:25:23.218 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66474.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66474.log.gz 2023-12-12T02:25:23.218 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66246.log 2023-12-12T02:25:23.219 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57143.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57143.log.gz 2023-12-12T02:25:23.219 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58228.log 2023-12-12T02:25:23.220 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66246.log: 91.8% -- replaced with /var/log/ceph/cluster1-client.mirror.2.75330.log.gz 2023-12-12T02:25:23.221 INFO:teuthology.orchestra.run.smithi140.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66246.log.gz 2023-12-12T02:25:23.221 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77559.log 2023-12-12T02:25:23.222 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58639.log 2023-12-12T02:25:23.223 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58228.log: /var/log/ceph/cluster2-client.mirror.77559.log: 87.8% 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58228.log.gz 2023-12-12T02:25:23.223 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.77559.log.gz 2023-12-12T02:25:23.223 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67321.log 2023-12-12T02:25:23.223 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79213.log 2023-12-12T02:25:23.224 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58639.log: /var/log/ceph/cluster2-client.mirror.67321.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58639.log.gz 2023-12-12T02:25:23.224 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67321.log.gz 2023-12-12T02:25:23.224 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42427.log 2023-12-12T02:25:23.225 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47213.log 2023-12-12T02:25:23.225 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79213.log: /var/log/ceph/cluster2-client.mirror.42427.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79213.log.gz 2023-12-12T02:25:23.225 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66660.log 2023-12-12T02:25:23.226 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47213.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42427.log.gz 2023-12-12T02:25:23.230 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47213.log.gz 2023-12-12T02:25:23.231 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70809.log 2023-12-12T02:25:23.231 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62266.log 2023-12-12T02:25:23.232 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66660.log: /var/log/ceph/cluster1-client.mirror.70809.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70809.log.gz 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66660.log.gz 2023-12-12T02:25:23.232 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.232 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80851.log 2023-12-12T02:25:23.233 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44745.log 2023-12-12T02:25:23.233 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62266.log: /var/log/ceph/cluster2-client.admin.80851.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80851.log.gz 2023-12-12T02:25:23.233 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62266.log.gz 2023-12-12T02:25:23.234 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58754.log 2023-12-12T02:25:23.234 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46816.log 2023-12-12T02:25:23.235 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44745.log: /var/log/ceph/cluster1-client.mirror.58754.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58754.log.gz 2023-12-12T02:25:23.239 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44745.log.gz 2023-12-12T02:25:23.239 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57744.log 2023-12-12T02:25:23.240 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67717.log 2023-12-12T02:25:23.240 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49089.log.gz 2023-12-12T02:25:23.241 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46816.log: /var/log/ceph/cluster1-client.mirror.57744.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46816.log.gz 2023-12-12T02:25:23.241 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57744.log.gz 2023-12-12T02:25:23.241 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63939.log 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71124.log 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67717.log: /var/log/ceph/cluster2-client.mirror.63939.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61187.log 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71124.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67717.log.gz 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63939.log.gz 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.71124.log.gz 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70314.log 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49881.log 2023-12-12T02:25:23.243 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61187.log: /var/log/ceph/cluster1-client.mirror.70314.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76582.log 2023-12-12T02:25:23.244 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70314.log.gz 2023-12-12T02:25:23.244 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49881.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.61187.log.gz 2023-12-12T02:25:23.245 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50332.log 2023-12-12T02:25:23.245 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39521.log 2023-12-12T02:25:23.245 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76582.log: /var/log/ceph/cluster2-client.mirror.50332.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76582.log.gz 2023-12-12T02:25:23.246 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42388.log 2023-12-12T02:25:23.246 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50332.log.gz 2023-12-12T02:25:23.247 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39521.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39521.log.gz 2023-12-12T02:25:23.247 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44476.log 2023-12-12T02:25:23.247 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73027.log 2023-12-12T02:25:23.248 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42388.log: /var/log/ceph/cluster2-client.mirror.44476.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.42388.log.gz 2023-12-12T02:25:23.248 INFO:teuthology.orchestra.run.smithi140.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44476.log.gz 2023-12-12T02:25:23.248 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73737.log 2023-12-12T02:25:23.250 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58205.log 2023-12-12T02:25:23.250 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73027.log: /var/log/ceph/cluster1-client.mirror.73737.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.73737.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.73027.log.gz 2023-12-12T02:25:23.250 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.250 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50374.log 2023-12-12T02:25:23.250 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44915.log 2023-12-12T02:25:23.251 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58205.log: /var/log/ceph/cluster2-client.mirror.50374.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58205.log.gz 2023-12-12T02:25:23.251 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62416.log 2023-12-12T02:25:23.253 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44915.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44915.log.gz 2023-12-12T02:25:23.253 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45260.log 2023-12-12T02:25:23.254 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62416.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62416.log.gz 2023-12-12T02:25:23.254 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63838.log 2023-12-12T02:25:23.255 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45260.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45260.log.gz 2023-12-12T02:25:23.255 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57721.log 2023-12-12T02:25:23.256 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63838.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63838.log.gz 2023-12-12T02:25:23.257 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56307.log 2023-12-12T02:25:23.258 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57721.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57721.log.gz 2023-12-12T02:25:23.258 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58708.log 2023-12-12T02:25:23.262 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56307.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56307.log.gz 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58619.log 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58708.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.58708.log.gz 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60533.log 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58619.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58619.log.gz 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49467.log 2023-12-12T02:25:23.263 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60533.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60533.log.gz 2023-12-12T02:25:23.264 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59843.log 2023-12-12T02:25:23.265 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49467.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49467.log.gz 2023-12-12T02:25:23.265 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69481.log 2023-12-12T02:25:23.266 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59843.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59843.log.gz 2023-12-12T02:25:23.266 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41102.log 2023-12-12T02:25:23.267 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69481.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69481.log.gz 2023-12-12T02:25:23.267 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44166.log 2023-12-12T02:25:23.268 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41102.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41102.log.gz 2023-12-12T02:25:23.269 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53459.log 2023-12-12T02:25:23.270 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44166.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44166.log.gz 2023-12-12T02:25:23.270 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79239.log 2023-12-12T02:25:23.272 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53459.log: 95.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53459.log.gz 2023-12-12T02:25:23.273 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66184.log 2023-12-12T02:25:23.273 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79239.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79239.log.gz 2023-12-12T02:25:23.273 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46105.log 2023-12-12T02:25:23.275 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66184.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66184.log.gz 2023-12-12T02:25:23.276 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43213.log 2023-12-12T02:25:23.277 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46105.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46105.log.gz 2023-12-12T02:25:23.277 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52058.log 2023-12-12T02:25:23.278 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43213.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43213.log.gz 2023-12-12T02:25:23.278 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80357.log 2023-12-12T02:25:23.296 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62457.log 2023-12-12T02:25:23.296 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80357.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80357.log.gz 2023-12-12T02:25:23.307 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62890.log 2023-12-12T02:25:23.309 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62457.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62457.log.gz 2023-12-12T02:25:23.323 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58956.log 2023-12-12T02:25:23.325 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62890.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62890.log.gz 2023-12-12T02:25:23.331 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77517.log 2023-12-12T02:25:23.331 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58956.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58956.log.gz 2023-12-12T02:25:23.339 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56349.log 2023-12-12T02:25:23.339 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77517.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.77517.log.gz 2023-12-12T02:25:23.347 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57187.log 2023-12-12T02:25:23.347 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56349.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56349.log.gz 2023-12-12T02:25:23.355 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45819.log 2023-12-12T02:25:23.356 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57187.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57187.log.gz 2023-12-12T02:25:23.363 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69211.log 2023-12-12T02:25:23.364 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.45819.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.45819.log.gz 2023-12-12T02:25:23.364 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60173.log 2023-12-12T02:25:23.364 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69211.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73439.log.gz 2023-12-12T02:25:23.365 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69211.log.gz 2023-12-12T02:25:23.365 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66618.log 2023-12-12T02:25:23.366 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60173.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60173.log.gz 2023-12-12T02:25:23.366 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43492.log 2023-12-12T02:25:23.368 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66618.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66618.log.gz 2023-12-12T02:25:23.368 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.48784.log 2023-12-12T02:25:23.369 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43492.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43492.log.gz 2023-12-12T02:25:23.369 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50925.log 2023-12-12T02:25:23.376 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.48784.log: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59910.log 2023-12-12T02:25:23.377 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50925.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50925.log.gz 2023-12-12T02:25:23.377 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.1.48784.log.gz 2023-12-12T02:25:23.378 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65286.log 2023-12-12T02:25:23.380 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59910.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59910.log.gz 2023-12-12T02:25:23.380 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74240.log 2023-12-12T02:25:23.381 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65286.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65286.log.gz 2023-12-12T02:25:23.381 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66804.log 2023-12-12T02:25:23.381 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74240.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74240.log.gz 2023-12-12T02:25:23.382 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42202.log 2023-12-12T02:25:23.384 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66804.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66804.log.gz 2023-12-12T02:25:23.384 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58182.log 2023-12-12T02:25:23.385 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42202.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42202.log.gz 2023-12-12T02:25:23.385 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41201.log 2023-12-12T02:25:23.386 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58182.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58182.log.gz 2023-12-12T02:25:23.386 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73216.log 2023-12-12T02:25:23.399 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41201.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44364.log 2023-12-12T02:25:23.399 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73216.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73216.log.gz 2023-12-12T02:25:23.415 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74727.log 2023-12-12T02:25:23.415 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44364.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44364.log.gz 2023-12-12T02:25:23.416 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59777.log 2023-12-12T02:25:23.430 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74727.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41201.log.gz 2023-12-12T02:25:23.435 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74075.log 2023-12-12T02:25:23.436 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59777.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59777.log.gz 2023-12-12T02:25:23.436 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54587.log 2023-12-12T02:25:23.437 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74075.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74075.log.gz 2023-12-12T02:25:23.451 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54791.log 2023-12-12T02:25:23.453 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54587.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54587.log.gz 2023-12-12T02:25:23.453 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49926.log 2023-12-12T02:25:23.453 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54791.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54791.log.gz 2023-12-12T02:25:23.454 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48784.log 2023-12-12T02:25:23.455 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49926.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49926.log.gz 2023-12-12T02:25:23.471 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49047.log 2023-12-12T02:25:23.475 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48784.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48784.log.gz 2023-12-12T02:25:23.490 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49373.log.gz 2023-12-12T02:25:23.491 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.41983.log 2023-12-12T02:25:23.491 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49047.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80617.log 2023-12-12T02:25:23.492 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.41983.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49047.log.gz 2023-12-12T02:25:23.507 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66701.log 2023-12-12T02:25:23.508 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80617.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80617.log.gz 2023-12-12T02:25:23.513 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52058.log.gz 2023-12-12T02:25:23.515 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43648.log 2023-12-12T02:25:23.516 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66701.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66701.log.gz 2023-12-12T02:25:23.516 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40132.log 2023-12-12T02:25:23.522 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43648.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43648.log.gz 2023-12-12T02:25:23.522 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58775.log 2023-12-12T02:25:23.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40132.log: 82.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40132.log.gz 2023-12-12T02:25:23.523 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80331.log 2023-12-12T02:25:23.525 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58775.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50374.log.gz 2023-12-12T02:25:23.525 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77356.log.gz 2023-12-12T02:25:23.525 INFO:teuthology.orchestra.run.smithi140.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58775.log.gz 2023-12-12T02:25:23.526 INFO:teuthology.orchestra.run.smithi055.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2023-12-12T02:25:23.527 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62910.log 2023-12-12T02:25:23.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80331.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70831.log 2023-12-12T02:25:23.527 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80331.log.gz 2023-12-12T02:25:23.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69166.log 2023-12-12T02:25:23.527 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64656.log 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70831.log: /var/log/ceph/cluster1-client.mirror.69166.log: 86.9% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69166.log.gz 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70831.log.gz 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47961.log 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62910.log.gz 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65438.log 2023-12-12T02:25:23.528 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47961.log: /var/log/ceph/cluster1-client.mirror.64656.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59409.log 2023-12-12T02:25:23.531 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65438.log: 88.3% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64656.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47961.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65438.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48124.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69751.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48124.log: /var/log/ceph/cluster1-client.mirror.59409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59820.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69751.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59409.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48124.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69751.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62110.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42310.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59820.log: /var/log/ceph/cluster1-client.mirror.62110.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54118.log 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59820.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62110.log.gz 2023-12-12T02:25:23.532 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44767.log 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79551.log 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54118.log: /var/log/ceph/cluster2-client.mirror.44767.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44767.log.gz 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77816.log 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54118.log.gz 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79551.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79551.log.gz 2023-12-12T02:25:23.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67094.log 2023-12-12T02:25:23.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67052.log 2023-12-12T02:25:23.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77816.log: /var/log/ceph/cluster2-client.mirror.67094.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77816.log.gz 2023-12-12T02:25:23.535 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67094.log.gz 2023-12-12T02:25:23.536 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56918.log 2023-12-12T02:25:23.536 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73134.log 2023-12-12T02:25:23.536 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67052.log: /var/log/ceph/cluster1-client.mirror.56918.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56918.log.gz 2023-12-12T02:25:23.537 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67052.log.gz 2023-12-12T02:25:23.537 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80539.log 2023-12-12T02:25:23.539 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51525.log 2023-12-12T02:25:23.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73134.log: /var/log/ceph/cluster2-client.admin.80539.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80539.log.gz 2023-12-12T02:25:23.539 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73134.log.gz 2023-12-12T02:25:23.539 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64903.log 2023-12-12T02:25:23.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60781.log 2023-12-12T02:25:23.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51525.log: /var/log/ceph/cluster2-client.mirror.64903.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51525.log.gz 2023-12-12T02:25:23.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67528.log 2023-12-12T02:25:23.541 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64903.log.gz 2023-12-12T02:25:23.541 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60781.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60781.log.gz 2023-12-12T02:25:23.541 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68850.log 2023-12-12T02:25:23.541 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53058.log 2023-12-12T02:25:23.542 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67528.log: /var/log/ceph/cluster2-client.mirror.68850.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67528.log.gz 2023-12-12T02:25:23.542 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50668.log 2023-12-12T02:25:23.543 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53058.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68850.log.gz 2023-12-12T02:25:23.551 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40025.log 2023-12-12T02:25:23.552 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50668.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50668.log.gz 2023-12-12T02:25:23.552 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76664.log 2023-12-12T02:25:23.553 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.40025.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40025.log.gz 2023-12-12T02:25:23.553 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80227.log 2023-12-12T02:25:23.554 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76664.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76664.log.gz 2023-12-12T02:25:23.554 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70449.log 2023-12-12T02:25:23.555 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80227.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80227.log.gz 2023-12-12T02:25:23.555 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69124.log 2023-12-12T02:25:23.556 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70449.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70449.log.gz 2023-12-12T02:25:23.557 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60620.log 2023-12-12T02:25:23.557 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69124.log: 5.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69124.log.gz 2023-12-12T02:25:23.557 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52547.log 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60620.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60620.log.gz 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51217.log 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52547.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52547.log.gz 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42090.log 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51217.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51217.log.gz 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74153.log 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42090.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.41983.log.gz 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42090.log.gz 2023-12-12T02:25:23.572 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80175.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74153.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74355.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7%/var/log/ceph/cluster2-client.admin.80175.log: -- replaced with /var/log/ceph/cluster2-client.mirror.74153.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80175.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65631.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70561.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74355.log: /var/log/ceph/cluster2-client.mirror.65631.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74355.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52979.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65631.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70561.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70561.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72408.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70381.log 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52979.log: /var/log/ceph/cluster1-client.mirror.72408.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72408.log.gz 2023-12-12T02:25:23.573 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74727.log.gz 2023-12-12T02:25:23.584 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65307.log 2023-12-12T02:25:23.584 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46681.log 2023-12-12T02:25:23.584 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70381.log: /var/log/ceph/cluster2-client.mirror.65307.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70381.log.gz 2023-12-12T02:25:23.584 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47623.log 2023-12-12T02:25:23.585 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46681.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65307.log.gz 2023-12-12T02:25:23.589 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46681.log.gz 2023-12-12T02:25:23.590 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62224.log 2023-12-12T02:25:23.590 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65415.log 2023-12-12T02:25:23.592 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62224.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62224.log.gz 2023-12-12T02:25:23.592 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78142.log 2023-12-12T02:25:23.593 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65415.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65415.log.gz 2023-12-12T02:25:23.593 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68083.log 2023-12-12T02:25:23.594 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.78142.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78142.log.gz 2023-12-12T02:25:23.594 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58398.log 2023-12-12T02:25:23.595 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68083.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68083.log.gz 2023-12-12T02:25:23.596 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.72704.log 2023-12-12T02:25:23.596 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47623.log.gz 2023-12-12T02:25:23.596 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58398.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.58398.log.gz 2023-12-12T02:25:23.597 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50315.log 2023-12-12T02:25:23.597 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.72704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61267.log 2023-12-12T02:25:23.598 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50315.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50315.log.gz 2023-12-12T02:25:23.598 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74482.log 2023-12-12T02:25:23.599 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61267.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61267.log.gz 2023-12-12T02:25:23.599 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48286.log 2023-12-12T02:25:23.600 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74482.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74482.log.gz 2023-12-12T02:25:23.601 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70359.log 2023-12-12T02:25:23.601 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48286.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48286.log.gz 2023-12-12T02:25:23.602 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46885.log 2023-12-12T02:25:23.602 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70359.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.70359.log.gz 2023-12-12T02:25:23.602 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74826.log 2023-12-12T02:25:23.603 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46885.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46885.log.gz 2023-12-12T02:25:23.604 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39611.log 2023-12-12T02:25:23.604 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74826.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74826.log.gz 2023-12-12T02:25:23.605 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43782.log 2023-12-12T02:25:23.606 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39611.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39611.log.gz 2023-12-12T02:25:23.606 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66722.log 2023-12-12T02:25:23.606 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43782.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.72704.log.gz 2023-12-12T02:25:23.611 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43782.log.gz 2023-12-12T02:25:23.611 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64346.log 2023-12-12T02:25:23.612 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2023-12-12T02:25:23.612 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66722.log: /var/log/ceph/cluster1-client.mirror.64346.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66722.log.gz 2023-12-12T02:25:23.613 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64346.log.gz 2023-12-12T02:25:23.613 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73298.log 2023-12-12T02:25:23.613 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41907.log 2023-12-12T02:25:23.614 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2.audit.log: /var/log/ceph/cluster2-client.mirror.73298.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73298.log.gz 2023-12-12T02:25:23.616 INFO:teuthology.orchestra.run.smithi140.stderr: 92.9% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2023-12-12T02:25:23.616 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50025.log 2023-12-12T02:25:23.617 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54497.log 2023-12-12T02:25:23.618 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.41907.log: /var/log/ceph/cluster2-client.mirror.50025.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41907.log.gz 2023-12-12T02:25:23.618 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50025.log.gz 2023-12-12T02:25:23.618 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62998.log 2023-12-12T02:25:23.619 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52095.log 2023-12-12T02:25:23.619 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54497.log: /var/log/ceph/cluster2-client.mirror.62998.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54497.log.gz 2023-12-12T02:25:23.619 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62998.log.gz 2023-12-12T02:25:23.619 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74397.log 2023-12-12T02:25:23.620 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71079.log 2023-12-12T02:25:23.620 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52095.log: /var/log/ceph/cluster1-client.mirror.74397.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74397.log.gz 2023-12-12T02:25:23.620 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52095.log.gz 2023-12-12T02:25:23.621 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59735.log 2023-12-12T02:25:23.621 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67300.log 2023-12-12T02:25:23.621 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71079.log: /var/log/ceph/cluster2-client.mirror.59735.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71079.log.gz 2023-12-12T02:25:23.622 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73195.log 2023-12-12T02:25:23.622 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59735.log.gz 2023-12-12T02:25:23.623 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67300.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67300.log.gz 2023-12-12T02:25:23.624 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56872.log 2023-12-12T02:25:23.624 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65530.log 2023-12-12T02:25:23.625 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73195.log: /var/log/ceph/cluster1-client.mirror.56872.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73195.log.gz 2023-12-12T02:25:23.625 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56872.log.gz 2023-12-12T02:25:23.625 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61508.log 2023-12-12T02:25:23.626 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60918.log 2023-12-12T02:25:23.626 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65530.log: /var/log/ceph/cluster2-client.mirror.61508.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65530.log.gz 2023-12-12T02:25:23.626 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.61508.log.gz 2023-12-12T02:25:23.626 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67888.log 2023-12-12T02:25:23.627 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48041.log 2023-12-12T02:25:23.627 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60918.log: /var/log/ceph/cluster2-client.mirror.67888.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60918.log.gz 2023-12-12T02:25:23.628 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66164.log 2023-12-12T02:25:23.629 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48041.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67888.log.gz 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48041.log.gz 2023-12-12T02:25:23.629 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.629 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39950.log 2023-12-12T02:25:23.630 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57031.log 2023-12-12T02:25:23.630 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66164.log: /var/log/ceph/cluster2-client.admin.39950.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39950.log.gz 2023-12-12T02:25:23.630 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66164.log.gz 2023-12-12T02:25:23.630 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78563.log 2023-12-12T02:25:23.631 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76393.log 2023-12-12T02:25:23.631 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57031.log: /var/log/ceph/cluster2-client.admin.78563.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78563.log.gz 2023-12-12T02:25:23.631 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57031.log.gz 2023-12-12T02:25:23.632 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56044.log 2023-12-12T02:25:23.632 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78901.log 2023-12-12T02:25:23.632 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76393.log: /var/log/ceph/cluster2-client.mirror.56044.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76393.log.gz 2023-12-12T02:25:23.632 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40514.log 2023-12-12T02:25:23.633 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78901.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78901.log.gz 2023-12-12T02:25:23.634 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose 92.4% -- /var/log/ceph/cluster1-client.mirror.61089.log -- replaced with /var/log/ceph/cluster2-client.mirror.56044.log.gz 2023-12-12T02:25:23.634 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:23.634 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40514.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40514.log.gz 2023-12-12T02:25:23.635 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61043.log 2023-12-12T02:25:23.635 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66308.log 2023-12-12T02:25:23.636 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61089.log: /var/log/ceph/cluster2-client.mirror.61043.log: 87.7% 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61089.log.gz 2023-12-12T02:25:23.636 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.61043.log.gz 2023-12-12T02:25:23.636 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45532.log 2023-12-12T02:25:23.637 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.76896.log 2023-12-12T02:25:23.638 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66308.log: /var/log/ceph/cluster2-client.mirror.45532.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45532.log.gz 2023-12-12T02:25:23.638 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66308.log.gz 2023-12-12T02:25:23.638 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2023-12-12T02:25:23.639 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66267.log 2023-12-12T02:25:23.640 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.3.76896.log: /var/log/ceph/cluster2.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.3.76896.log.gz 2023-12-12T02:25:23.641 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45644.log 2023-12-12T02:25:23.642 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66267.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66267.log.gz 2023-12-12T02:25:23.642 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79759.log 2023-12-12T02:25:23.643 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45644.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45644.log.gz 2023-12-12T02:25:23.645 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53365.log 2023-12-12T02:25:23.645 INFO:teuthology.orchestra.run.smithi140.stderr: 88.0% -- replaced with /var/log/ceph/cluster2.log.gz 2023-12-12T02:25:23.645 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79759.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79759.log.gz 2023-12-12T02:25:23.645 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39800.log 2023-12-12T02:25:23.645 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75208.log 2023-12-12T02:25:23.646 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53365.log: /var/log/ceph/cluster2-client.admin.39800.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39800.log.gz 2023-12-12T02:25:23.659 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44569.log 2023-12-12T02:25:23.660 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75208.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75208.log.gz 2023-12-12T02:25:23.661 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51821.log 2023-12-12T02:25:23.662 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44569.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44569.log.gz 2023-12-12T02:25:23.662 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49271.log 2023-12-12T02:25:23.674 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51821.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49881.log.gz 2023-12-12T02:25:23.674 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48601.log 2023-12-12T02:25:23.675 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49271.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49271.log.gz 2023-12-12T02:25:23.676 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.47058.log 2023-12-12T02:25:23.683 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48601.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45598.log 2023-12-12T02:25:23.715 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.47058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50150.log 2023-12-12T02:25:23.715 INFO:teuthology.orchestra.run.smithi140.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.47058.log.gz 2023-12-12T02:25:23.716 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8%/var/log/ceph/cluster1-client.mirror.45598.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53058.log.gz 2023-12-12T02:25:23.716 INFO:teuthology.orchestra.run.smithi140.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.45598.log.gz 2023-12-12T02:25:23.719 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40463.log 2023-12-12T02:25:23.719 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50150.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.50150.log.gz 2023-12-12T02:25:23.720 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.45427.log 2023-12-12T02:25:23.720 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40463.log: 53.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40463.log.gz 2023-12-12T02:25:23.721 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45795.log 2023-12-12T02:25:23.722 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.45427.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.45427.log.gz 2023-12-12T02:25:23.722 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51160.log 2023-12-12T02:25:23.739 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.46401.log 2023-12-12T02:25:23.747 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51160.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56760.log 2023-12-12T02:25:23.748 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.46401.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.46401.log.gz 2023-12-12T02:25:23.759 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45621.log 2023-12-12T02:25:23.759 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56760.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56760.log.gz 2023-12-12T02:25:23.777 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52979.log.gz 2023-12-12T02:25:23.779 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67424.log 2023-12-12T02:25:23.780 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45621.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45621.log.gz 2023-12-12T02:25:23.791 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51878.log 2023-12-12T02:25:23.793 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67424.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67424.log.gz 2023-12-12T02:25:23.803 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49509.log 2023-12-12T02:25:23.803 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51878.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51878.log.gz 2023-12-12T02:25:23.819 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73902.log 2023-12-12T02:25:23.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49509.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49595.log 2023-12-12T02:25:23.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73902.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73902.log.gz 2023-12-12T02:25:23.847 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62324.log 2023-12-12T02:25:23.855 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49595.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67610.log 2023-12-12T02:25:23.856 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62324.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62324.log.gz 2023-12-12T02:25:23.868 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64074.log 2023-12-12T02:25:23.870 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67610.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67610.log.gz 2023-12-12T02:25:23.881 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51584.log 2023-12-12T02:25:23.882 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64074.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64074.log.gz 2023-12-12T02:25:23.891 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67466.log 2023-12-12T02:25:23.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51584.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79343.log 2023-12-12T02:25:23.911 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67466.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67466.log.gz 2023-12-12T02:25:23.927 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70066.log 2023-12-12T02:25:23.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79343.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79343.log.gz 2023-12-12T02:25:23.947 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67847.log 2023-12-12T02:25:23.947 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70066.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70066.log.gz 2023-12-12T02:25:23.963 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72496.log 2023-12-12T02:25:23.964 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67847.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67847.log.gz 2023-12-12T02:25:23.979 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59169.log 2023-12-12T02:25:23.979 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.72496.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.72496.log.gz 2023-12-12T02:25:23.990 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42310.log.gz 2023-12-12T02:25:23.995 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43300.log 2023-12-12T02:25:23.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59169.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59169.log.gz 2023-12-12T02:25:24.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42246.log 2023-12-12T02:25:24.023 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43300.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43300.log.gz 2023-12-12T02:25:24.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51197.log 2023-12-12T02:25:24.028 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42246.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42246.log.gz 2023-12-12T02:25:24.039 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65194.log 2023-12-12T02:25:24.039 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51197.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51197.log.gz 2023-12-12T02:25:24.045 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53365.log.gz 2023-12-12T02:25:24.051 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70291.log 2023-12-12T02:25:24.051 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65194.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65194.log.gz 2023-12-12T02:25:24.064 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51821.log.gz 2023-12-12T02:25:24.067 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76167.log 2023-12-12T02:25:24.067 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70291.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70291.log.gz 2023-12-12T02:25:24.086 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51920.log 2023-12-12T02:25:24.086 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76167.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76167.log.gz 2023-12-12T02:25:24.095 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67218.log 2023-12-12T02:25:24.095 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49509.log.gz/var/log/ceph/cluster2-client.mirror.51920.log: 2023-12-12T02:25:24.095 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51920.log.gz 2023-12-12T02:25:24.099 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51542.log 2023-12-12T02:25:24.100 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67218.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67218.log.gz 2023-12-12T02:25:24.100 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42603.log 2023-12-12T02:25:24.101 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51542.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51542.log.gz 2023-12-12T02:25:24.102 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63723.log 2023-12-12T02:25:24.107 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42603.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42603.log.gz 2023-12-12T02:25:24.107 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45125.log 2023-12-12T02:25:24.108 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63723.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63723.log.gz 2023-12-12T02:25:24.109 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51012.log 2023-12-12T02:25:24.110 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45125.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45125.log.gz 2023-12-12T02:25:24.110 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75589.log 2023-12-12T02:25:24.111 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51012.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51012.log.gz 2023-12-12T02:25:24.111 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54138.log 2023-12-12T02:25:24.112 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75589.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75589.log.gz 2023-12-12T02:25:24.112 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66680.log 2023-12-12T02:25:24.115 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54138.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54138.log.gz 2023-12-12T02:25:24.115 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45986.log 2023-12-12T02:25:24.117 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66680.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66680.log.gz 2023-12-12T02:25:24.117 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73069.log 2023-12-12T02:25:24.122 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45986.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45986.log.gz 2023-12-12T02:25:24.122 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67868.log 2023-12-12T02:25:24.123 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73069.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.73069.log.gz 2023-12-12T02:25:24.123 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.40204.log 2023-12-12T02:25:24.124 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67868.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67868.log.gz 2023-12-12T02:25:24.124 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34910.log 2023-12-12T02:25:24.138 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.3.40204.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45795.log.gz 2023-12-12T02:25:24.138 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.40204.log.gz 2023-12-12T02:25:24.139 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48328.log 2023-12-12T02:25:24.139 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64005.log 2023-12-12T02:25:24.139 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.34910.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34910.log.gz 2023-12-12T02:25:24.140 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.48328.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60110.log 2023-12-12T02:25:24.140 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48328.log.gz 2023-12-12T02:25:24.140 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64005.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64005.log.gz 2023-12-12T02:25:24.141 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57488.log 2023-12-12T02:25:24.141 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73004.log 2023-12-12T02:25:24.142 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60110.log: /var/log/ceph/cluster2-client.mirror.57488.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60110.log.gz 2023-12-12T02:25:24.142 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67032.log 2023-12-12T02:25:24.142 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57488.log.gz 2023-12-12T02:25:24.143 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73004.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73004.log.gz 2023-12-12T02:25:24.143 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67135.log 2023-12-12T02:25:24.144 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74565.log 2023-12-12T02:25:24.144 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67032.log: /var/log/ceph/cluster2-client.mirror.67135.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67032.log.gz 2023-12-12T02:25:24.144 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67135.log.gz 2023-12-12T02:25:24.145 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47772.log 2023-12-12T02:25:24.145 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75558.log 2023-12-12T02:25:24.145 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74565.log: /var/log/ceph/cluster1-client.mirror.47772.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74565.log.gz 2023-12-12T02:25:24.146 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74627.log 2023-12-12T02:25:24.146 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75558.log: 93.8% 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75558.log.gz 2023-12-12T02:25:24.147 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49595.log.gz 2023-12-12T02:25:24.147 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49487.log 2023-12-12T02:25:24.147 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71371.log 2023-12-12T02:25:24.148 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74627.log: /var/log/ceph/cluster2-client.mirror.49487.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74627.log.gz 2023-12-12T02:25:24.148 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49487.log.gz 2023-12-12T02:25:24.148 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70179.log 2023-12-12T02:25:24.149 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50888.log 2023-12-12T02:25:24.149 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71371.log: /var/log/ceph/cluster1-client.mirror.70179.log: 86.9% 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71371.log.gz 2023-12-12T02:25:24.149 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.70179.log.gz 2023-12-12T02:25:24.149 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69189.log 2023-12-12T02:25:24.149 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52377.log 2023-12-12T02:25:24.151 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50888.log: /var/log/ceph/cluster1-client.mirror.69189.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50888.log.gz 2023-12-12T02:25:24.151 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69189.log.gz 2023-12-12T02:25:24.151 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45145.log 2023-12-12T02:25:24.151 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47772.log.gz 2023-12-12T02:25:24.151 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52377.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52377.log.gz 2023-12-12T02:25:24.152 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60959.log 2023-12-12T02:25:24.152 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45145.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69661.log 2023-12-12T02:25:24.152 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60959.log: 88.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68830.log 2023-12-12T02:25:24.152 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.45145.log.gz 2023-12-12T02:25:24.153 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70876.log 2023-12-12T02:25:24.154 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68830.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69661.log.gz 2023-12-12T02:25:24.154 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77452.log 2023-12-12T02:25:24.155 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68830.log.gz 2023-12-12T02:25:24.155 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60959.log.gz 2023-12-12T02:25:24.155 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70876.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.70876.log.gz 2023-12-12T02:25:24.155 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76870.log 2023-12-12T02:25:24.155 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42925.log 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62156.log 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76870.log: /var/log/ceph/cluster1-client.mirror.42925.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.77452.log.gz 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% 74.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42925.log.gz 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.76870.log.gz 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53343.log 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.73672.log 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62156.log: /var/log/ceph/cluster2-client.mirror.53343.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78021.log 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.73672.log: 87.7% 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53343.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.62156.log.gz 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.157 INFO:teuthology.orchestra.run.smithi140.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.73672.log.gz 2023-12-12T02:25:24.158 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68062.log 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75678.log 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.78021.log: /var/log/ceph/cluster2-client.mirror.68062.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76986.log 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78021.log.gz 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75678.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68062.log.gz 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39442.log 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75678.log.gz 2023-12-12T02:25:24.161 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39306.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58274.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76986.log: /var/log/ceph/cluster2-client.admin.39442.log: /var/log/ceph/cluster2-client.admin.39306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39442.log.gz 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.76986.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39306.log.gz 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47494.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58274.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41983.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47494.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58274.log.gz 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79993.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76561.log 2023-12-12T02:25:24.162 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.41983.log: /var/log/ceph/cluster2-client.admin.79993.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79993.log.gz 2023-12-12T02:25:24.164 INFO:teuthology.orchestra.run.smithi140.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41983.log.gz 2023-12-12T02:25:24.164 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72834.log 2023-12-12T02:25:24.165 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76561.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55637.log 2023-12-12T02:25:24.166 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76561.log.gz/var/log/ceph/cluster2-client.mirror.72834.log: 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47494.log.gz 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72834.log.gz 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68289.log 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61420.log 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55637.log: /var/log/ceph/cluster2-client.mirror.68289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55382.log 2023-12-12T02:25:24.167 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55637.log.gz 2023-12-12T02:25:24.168 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50809.log 2023-12-12T02:25:24.168 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55382.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68289.log.gz 2023-12-12T02:25:24.168 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.61420.log.gz 2023-12-12T02:25:24.169 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55382.log.gz 2023-12-12T02:25:24.169 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41961.log 2023-12-12T02:25:24.169 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48472.log 2023-12-12T02:25:24.169 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50809.log: /var/log/ceph/cluster2-client.mirror.41961.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78166.log 2023-12-12T02:25:24.170 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50809.log.gz 2023-12-12T02:25:24.171 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48472.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74668.log 2023-12-12T02:25:24.171 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41961.log.gz 2023-12-12T02:25:24.171 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78166.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48472.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78166.log.gz 2023-12-12T02:25:24.171 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.172 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42364.log 2023-12-12T02:25:24.172 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67486.log 2023-12-12T02:25:24.172 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74668.log.gz 2023-12-12T02:25:24.172 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60467.log 2023-12-12T02:25:24.172 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49293.log 2023-12-12T02:25:24.173 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42364.log.gz 2023-12-12T02:25:24.173 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67486.log: /var/log/ceph/cluster2-client.mirror.60467.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.60467.log.gz 2023-12-12T02:25:24.173 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67486.log.gz 2023-12-12T02:25:24.173 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48601.log.gz 2023-12-12T02:25:24.174 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39384.log 2023-12-12T02:25:24.174 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74647.log 2023-12-12T02:25:24.175 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68641.log 2023-12-12T02:25:24.175 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39384.log.gz 2023-12-12T02:25:24.175 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74647.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59363.log 2023-12-12T02:25:24.175 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45667.log 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68641.log: /var/log/ceph/cluster1-client.mirror.59363.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68641.log.gz 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.59363.log.gz 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74545.log 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74647.log.gz 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50047.log 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45667.log: /var/log/ceph/cluster2-client.mirror.74545.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54768.log 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50047.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74545.log.gz 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45667.log.gz 2023-12-12T02:25:24.177 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49251.log 2023-12-12T02:25:24.178 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51445.log 2023-12-12T02:25:24.179 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54768.log: /var/log/ceph/cluster2-client.mirror.49251.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54768.log.gz 2023-12-12T02:25:24.179 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47514.log 2023-12-12T02:25:24.179 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49251.log.gz 2023-12-12T02:25:24.179 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51445.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51445.log.gz 2023-12-12T02:25:24.180 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58662.log 2023-12-12T02:25:24.180 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55551.log 2023-12-12T02:25:24.180 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47514.log: /var/log/ceph/cluster1-client.mirror.58662.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58662.log.gz 2023-12-12T02:25:24.185 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47514.log.gz 2023-12-12T02:25:24.185 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66598.log 2023-12-12T02:25:24.186 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44615.log 2023-12-12T02:25:24.186 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55551.log: /var/log/ceph/cluster2-client.mirror.66598.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55551.log.gz 2023-12-12T02:25:24.186 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66598.log.gz 2023-12-12T02:25:24.186 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51957.log 2023-12-12T02:25:24.187 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59453.log 2023-12-12T02:25:24.188 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44615.log: /var/log/ceph/cluster2-client.mirror.51957.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44615.log.gz 2023-12-12T02:25:24.188 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51957.log.gz 2023-12-12T02:25:24.188 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61962.log 2023-12-12T02:25:24.189 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64633.log 2023-12-12T02:25:24.189 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59453.log: /var/log/ceph/cluster1-client.mirror.61962.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61962.log.gz 2023-12-12T02:25:24.190 INFO:teuthology.orchestra.run.smithi140.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59453.log.gz 2023-12-12T02:25:24.190 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48579.log 2023-12-12T02:25:24.190 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52606.log 2023-12-12T02:25:24.191 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64633.log: /var/log/ceph/cluster2-client.mirror.48579.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64633.log.gz 2023-12-12T02:25:24.191 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48579.log.gz 2023-12-12T02:25:24.191 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77112.log 2023-12-12T02:25:24.192 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51562.log 2023-12-12T02:25:24.193 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52606.log: /var/log/ceph/cluster2-client.mirror.77112.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77112.log.gz 2023-12-12T02:25:24.193 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52606.log.gz 2023-12-12T02:25:24.193 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61735.log 2023-12-12T02:25:24.193 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58573.log 2023-12-12T02:25:24.194 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51562.log: /var/log/ceph/cluster2-client.mirror.61735.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51562.log.gz 2023-12-12T02:25:24.194 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73113.log 2023-12-12T02:25:24.195 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58573.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58573.log.gz 2023-12-12T02:25:24.195 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61735.log.gz 2023-12-12T02:25:24.196 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71191.log 2023-12-12T02:25:24.196 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73113.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.73113.log.gz 2023-12-12T02:25:24.196 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65854.log 2023-12-12T02:25:24.197 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57621.log 2023-12-12T02:25:24.197 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71191.log: /var/log/ceph/cluster2-client.mirror.65854.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71191.log.gz 2023-12-12T02:25:24.198 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65854.log.gz 2023-12-12T02:25:24.198 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44526.log 2023-12-12T02:25:24.199 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57621.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57621.log.gz 2023-12-12T02:25:24.200 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52137.log 2023-12-12T02:25:24.200 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76763.log 2023-12-12T02:25:24.201 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44526.log: /var/log/ceph/cluster2-client.mirror.52137.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44526.log.gz 2023-12-12T02:25:24.201 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45895.log 2023-12-12T02:25:24.202 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76763.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76763.log.gz 2023-12-12T02:25:24.203 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40975.log 2023-12-12T02:25:24.206 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45895.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51160.log.gz 2023-12-12T02:25:24.211 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74462.log 2023-12-12T02:25:24.211 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40975.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40975.log.gz 2023-12-12T02:25:24.212 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52858.log 2023-12-12T02:25:24.213 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74462.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.74462.log.gz 2023-12-12T02:25:24.213 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69369.log 2023-12-12T02:25:24.214 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52858.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52858.log.gz 2023-12-12T02:25:24.215 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56392.log 2023-12-12T02:25:24.215 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69369.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69369.log.gz 2023-12-12T02:25:24.216 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63304.log 2023-12-12T02:25:24.216 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56392.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56392.log.gz 2023-12-12T02:25:24.217 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80305.log 2023-12-12T02:25:24.218 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63304.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63304.log.gz 2023-12-12T02:25:24.218 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.40132.log 2023-12-12T02:25:24.219 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80305.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80305.log.gz 2023-12-12T02:25:24.219 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59952.log 2023-12-12T02:25:24.220 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.40132.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51584.log.gz 2023-12-12T02:25:24.220 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78433.log 2023-12-12T02:25:24.222 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59952.log: 91.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.40132.log.gz 2023-12-12T02:25:24.222 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76327.log 2023-12-12T02:25:24.222 INFO:teuthology.orchestra.run.smithi140.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59952.log.gz 2023-12-12T02:25:24.222 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78433.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78433.log.gz 2023-12-12T02:25:24.222 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47342.log 2023-12-12T02:25:24.223 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78329.log 2023-12-12T02:25:24.223 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.76327.log.gz/var/log/ceph/cluster1-client.mirror.47342.log: 2023-12-12T02:25:24.223 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47643.log 2023-12-12T02:25:24.224 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78329.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78329.log.gz 2023-12-12T02:25:24.224 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74912.log 2023-12-12T02:25:24.228 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47643.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47342.log.gz 2023-12-12T02:25:24.229 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78277.log 2023-12-12T02:25:24.229 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74912.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74912.log.gz 2023-12-12T02:25:24.230 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47643.log.gz 2023-12-12T02:25:24.230 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69526.log 2023-12-12T02:25:24.231 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78277.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78277.log.gz 2023-12-12T02:25:24.231 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61870.log 2023-12-12T02:25:24.232 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69526.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71146.log 2023-12-12T02:25:24.232 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69526.log.gz 2023-12-12T02:25:24.232 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61870.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46194.log 2023-12-12T02:25:24.232 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61870.log.gz 2023-12-12T02:25:24.232 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71146.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71146.log.gz 2023-12-12T02:25:24.233 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47901.log 2023-12-12T02:25:24.234 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46194.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70516.log 2023-12-12T02:25:24.235 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47901.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46194.log.gz 2023-12-12T02:25:24.235 INFO:teuthology.orchestra.run.smithi140.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53960.log 2023-12-12T02:25:24.235 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47901.log.gz 2023-12-12T02:25:24.235 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70516.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70516.log.gz 2023-12-12T02:25:24.235 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60399.log 2023-12-12T02:25:24.236 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53960.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75778.log 2023-12-12T02:25:24.236 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.60399.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53960.log.gz 2023-12-12T02:25:24.236 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73758.log 2023-12-12T02:25:24.237 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75778.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75778.log.gz 2023-12-12T02:25:24.237 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.60399.log.gz 2023-12-12T02:25:24.237 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54543.log 2023-12-12T02:25:24.238 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73758.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73758.log.gz 2023-12-12T02:25:24.238 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79967.log 2023-12-12T02:25:24.239 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54543.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64278.log 2023-12-12T02:25:24.239 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54543.log.gz 2023-12-12T02:25:24.239 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79967.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79967.log.gz 2023-12-12T02:25:24.239 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79525.log 2023-12-12T02:25:24.240 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64278.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49822.log 2023-12-12T02:25:24.240 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79525.log: 88.9% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79525.log.gz 2023-12-12T02:25:24.240 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.64278.log.gz 2023-12-12T02:25:24.241 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65395.log 2023-12-12T02:25:24.241 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72892.log 2023-12-12T02:25:24.241 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49822.log: /var/log/ceph/cluster2-client.mirror.65395.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49822.log.gz 2023-12-12T02:25:24.242 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74870.log 2023-12-12T02:25:24.242 INFO:teuthology.orchestra.run.smithi140.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65395.log.gz 2023-12-12T02:25:24.243 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72892.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.72892.log.gz 2023-12-12T02:25:24.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67280.log 2023-12-12T02:25:24.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46839.log 2023-12-12T02:25:24.244 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74870.log: /var/log/ceph/cluster2-client.mirror.67280.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74870.log.gz 2023-12-12T02:25:24.244 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.67280.log.gz 2023-12-12T02:25:24.244 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52278.log 2023-12-12T02:25:24.245 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46839.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46839.log.gz 2023-12-12T02:25:24.245 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69841.log 2023-12-12T02:25:24.246 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54241.log 2023-12-12T02:25:24.246 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52278.log: /var/log/ceph/cluster1-client.mirror.69841.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69841.log.gz 2023-12-12T02:25:24.247 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52278.log.gz 2023-12-12T02:25:24.247 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54331.log 2023-12-12T02:25:24.247 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77334.log 2023-12-12T02:25:24.247 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54241.log: /var/log/ceph/cluster2-client.mirror.54331.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54241.log.gz 2023-12-12T02:25:24.248 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.55249.log 2023-12-12T02:25:24.248 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77334.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77334.log.gz 2023-12-12T02:25:24.249 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49988.log 2023-12-12T02:25:24.249 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54331.log.gz 2023-12-12T02:25:24.255 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.2.55249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79733.log 2023-12-12T02:25:24.255 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49988.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49988.log.gz 2023-12-12T02:25:24.256 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70921.log 2023-12-12T02:25:24.256 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79733.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79733.log.gz 2023-12-12T02:25:24.257 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47686.log 2023-12-12T02:25:24.257 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70921.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70921.log.gz 2023-12-12T02:25:24.258 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56651.log 2023-12-12T02:25:24.260 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47686.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45895.log.gz 2023-12-12T02:25:24.263 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47686.log.gz 2023-12-12T02:25:24.264 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55574.log 2023-12-12T02:25:24.264 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48393.log 2023-12-12T02:25:24.265 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56651.log: /var/log/ceph/cluster1-client.mirror.55574.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56651.log.gz 2023-12-12T02:25:24.266 INFO:teuthology.orchestra.run.smithi140.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55574.log.gz 2023-12-12T02:25:24.266 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46505.log 2023-12-12T02:25:24.266 INFO:teuthology.orchestra.run.smithi140.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.2.55249.log.gz 2023-12-12T02:25:24.266 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53036.log 2023-12-12T02:25:24.266 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48393.log: /var/log/ceph/cluster2-client.mirror.46505.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80747.log 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53036.log: 89.6% 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48393.log.gz 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53036.log.gz 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.46505.log.gz 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59103.log 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46617.log 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80747.log: /var/log/ceph/cluster2-client.mirror.59103.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75443.log 2023-12-12T02:25:24.268 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80747.log.gz 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46617.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69459.log 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59103.log.gz 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46617.log.gz 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75443.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75443.log.gz 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.44004.log 2023-12-12T02:25:24.269 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51720.log 2023-12-12T02:25:24.270 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69459.log: /var/log/ceph/cluster1-client.mirror.0.44004.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53549.log 2023-12-12T02:25:24.271 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51720.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69459.log.gz 2023-12-12T02:25:24.271 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43320.log 2023-12-12T02:25:24.271 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51720.log.gz 2023-12-12T02:25:24.271 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53549.log: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53549.log.gz 2023-12-12T02:25:24.271 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68809.log 2023-12-12T02:25:24.272 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59320.log 2023-12-12T02:25:24.272 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43320.log: /var/log/ceph/cluster2-client.mirror.68809.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68809.log.gz 2023-12-12T02:25:24.275 INFO:teuthology.orchestra.run.smithi140.stderr: 92.8% -- replaced with /var/log/ceph/cluster1-client.mirror.0.44004.log.gz 2023-12-12T02:25:24.277 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43320.log.gz 2023-12-12T02:25:24.277 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62932.log 2023-12-12T02:25:24.278 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42744.log 2023-12-12T02:25:24.278 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59320.log: /var/log/ceph/cluster2-client.mirror.62932.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79369.log 2023-12-12T02:25:24.280 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42744.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62932.log.gz 2023-12-12T02:25:24.280 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.59320.log.gz 2023-12-12T02:25:24.280 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42744.log.gz 2023-12-12T02:25:24.280 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52589.log 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.40197.log 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79369.log: /var/log/ceph/cluster2-client.mirror.52589.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78381.log 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79369.log.gz 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9%/var/log/ceph/cluster2-client.mirror.2.40197.log: -- replaced with /var/log/ceph/cluster2-client.mirror.52589.log.gz 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60131.log 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75035.log 2023-12-12T02:25:24.281 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78381.log: /var/log/ceph/cluster2-client.mirror.60131.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78381.log.gz 2023-12-12T02:25:24.282 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79629.log 2023-12-12T02:25:24.282 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75035.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75035.log.gz 2023-12-12T02:25:24.282 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60131.log.gz 2023-12-12T02:25:24.282 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71490.log 2023-12-12T02:25:24.283 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45873.log 2023-12-12T02:25:24.283 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79629.log: /var/log/ceph/cluster2-client.mirror.71490.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79629.log.gz 2023-12-12T02:25:24.283 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54629.log 2023-12-12T02:25:24.283 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.71490.log.gz 2023-12-12T02:25:24.284 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45873.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45873.log.gz 2023-12-12T02:25:24.284 INFO:teuthology.orchestra.run.smithi140.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.2.40197.log.gz 2023-12-12T02:25:24.284 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45324.log 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61668.log 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54629.log: /var/log/ceph/cluster2-client.mirror.45324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49450.log 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61668.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54629.log.gz 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61668.log.gz 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77276.log 2023-12-12T02:25:24.286 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49450.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49450.log.gz 2023-12-12T02:25:24.287 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79057.log 2023-12-12T02:25:24.288 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64137.log 2023-12-12T02:25:24.288 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77276.log: /var/log/ceph/cluster2-client.admin.79057.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79057.log.gz 2023-12-12T02:25:24.288 INFO:teuthology.orchestra.run.smithi140.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.77276.log.gz 2023-12-12T02:25:24.288 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50868.log 2023-12-12T02:25:24.289 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62370.log 2023-12-12T02:25:24.289 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64137.log: /var/log/ceph/cluster2-client.mirror.50868.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64137.log.gz 2023-12-12T02:25:24.290 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61400.log 2023-12-12T02:25:24.290 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45324.log.gz 2023-12-12T02:25:24.291 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62370.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62370.log.gz 2023-12-12T02:25:24.291 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39674.log 2023-12-12T02:25:24.291 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45708.log 2023-12-12T02:25:24.292 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61400.log: /var/log/ceph/cluster2-client.admin.39674.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39674.log.gz 2023-12-12T02:25:24.293 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61400.log.gz 2023-12-12T02:25:24.294 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57360.log 2023-12-12T02:25:24.294 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43277.log 2023-12-12T02:25:24.294 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45708.log: /var/log/ceph/cluster1-client.mirror.57360.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57360.log.gz 2023-12-12T02:25:24.299 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45708.log.gz 2023-12-12T02:25:24.299 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64051.log 2023-12-12T02:25:24.300 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71304.log 2023-12-12T02:25:24.300 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43277.log: /var/log/ceph/cluster1-client.mirror.64051.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64051.log.gz 2023-12-12T02:25:24.305 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43277.log.gz 2023-12-12T02:25:24.306 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60510.log 2023-12-12T02:25:24.306 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43581.log 2023-12-12T02:25:24.306 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71304.log: /var/log/ceph/cluster1-client.mirror.60510.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.71304.log.gz 2023-12-12T02:25:24.307 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60510.log.gz 2023-12-12T02:25:24.307 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50130.log 2023-12-12T02:25:24.307 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43581.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43581.log.gz 2023-12-12T02:25:24.308 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78797.log 2023-12-12T02:25:24.308 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78693.log 2023-12-12T02:25:24.308 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50130.log: /var/log/ceph/cluster2-client.admin.78797.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78797.log.gz 2023-12-12T02:25:24.315 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40941.log 2023-12-12T02:25:24.315 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78693.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78693.log.gz 2023-12-12T02:25:24.316 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55361.log 2023-12-12T02:25:24.316 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40941.log: 56.2% -- replaced with /var/log/ceph/cluster2-client.mirror.40941.log.gz 2023-12-12T02:25:24.317 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72431.log 2023-12-12T02:25:24.318 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55361.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55361.log.gz 2023-12-12T02:25:24.318 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79135.log 2023-12-12T02:25:24.319 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.72431.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.72431.log.gz 2023-12-12T02:25:24.319 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60421.log 2023-12-12T02:25:24.320 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79135.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79135.log.gz 2023-12-12T02:25:24.320 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44342.log 2023-12-12T02:25:24.321 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60421.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60421.log.gz 2023-12-12T02:25:24.322 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52668.log 2023-12-12T02:25:24.327 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44342.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44342.log.gz 2023-12-12T02:25:24.328 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39358.log 2023-12-12T02:25:24.328 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52668.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52668.log.gz 2023-12-12T02:25:24.329 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65766.log 2023-12-12T02:25:24.329 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39358.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39358.log.gz 2023-12-12T02:25:24.329 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71349.log 2023-12-12T02:25:24.330 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65766.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65766.log.gz 2023-12-12T02:25:24.331 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54376.log 2023-12-12T02:25:24.332 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71349.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71349.log.gz 2023-12-12T02:25:24.332 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43167.log 2023-12-12T02:25:24.333 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54376.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54376.log.gz 2023-12-12T02:25:24.333 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62304.log 2023-12-12T02:25:24.334 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43167.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43167.log.gz 2023-12-12T02:25:24.334 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.40088.log 2023-12-12T02:25:24.336 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62304.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.62304.log.gz 2023-12-12T02:25:24.336 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67156.log 2023-12-12T02:25:24.347 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.3.40088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63635.log 2023-12-12T02:25:24.348 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67156.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67156.log.gz 2023-12-12T02:25:24.363 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58078.log 2023-12-12T02:25:24.365 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63635.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63635.log.gz 2023-12-12T02:25:24.379 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43190.log 2023-12-12T02:25:24.380 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58078.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58078.log.gz 2023-12-12T02:25:24.387 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78771.log 2023-12-12T02:25:24.387 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43190.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43190.log.gz 2023-12-12T02:25:24.399 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43418.log 2023-12-12T02:25:24.399 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78771.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78771.log.gz 2023-12-12T02:25:24.407 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43958.log 2023-12-12T02:25:24.407 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43418.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43418.log.gz 2023-12-12T02:25:24.415 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75913.log 2023-12-12T02:25:24.415 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43958.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43958.log.gz 2023-12-12T02:25:24.423 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56438.log 2023-12-12T02:25:24.423 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75913.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.75913.log.gz 2023-12-12T02:25:24.431 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62754.log 2023-12-12T02:25:24.431 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56438.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56438.log.gz 2023-12-12T02:25:24.439 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47408.log 2023-12-12T02:25:24.439 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62754.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62754.log.gz 2023-12-12T02:25:24.442 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49293.log.gz 2023-12-12T02:25:24.443 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52137.log.gz 2023-12-12T02:25:24.447 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68103.log 2023-12-12T02:25:24.447 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47408.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68974.log 2023-12-12T02:25:24.449 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68103.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68103.log.gz 2023-12-12T02:25:24.449 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66494.log 2023-12-12T02:25:24.451 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68974.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68974.log.gz 2023-12-12T02:25:24.451 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65936.log 2023-12-12T02:25:24.452 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66494.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47408.log.gz 2023-12-12T02:25:24.453 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66494.log.gz 2023-12-12T02:25:24.454 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41881.log 2023-12-12T02:25:24.454 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65651.log 2023-12-12T02:25:24.454 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65936.log: /var/log/ceph/cluster1-client.admin.41881.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.41881.log.gz 2023-12-12T02:25:24.455 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65936.log.gz 2023-12-12T02:25:24.455 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40915.log 2023-12-12T02:25:24.456 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47752.log 2023-12-12T02:25:24.456 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65651.log: /var/log/ceph/cluster2-client.admin.40915.log: 67.4% -- replaced with /var/log/ceph/cluster2-client.admin.40915.log.gz 2023-12-12T02:25:24.456 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65651.log.gz 2023-12-12T02:25:24.457 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46640.log 2023-12-12T02:25:24.457 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51275.log 2023-12-12T02:25:24.458 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47752.log: /var/log/ceph/cluster1-client.mirror.46640.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46640.log.gz 2023-12-12T02:25:24.462 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47752.log.gz 2023-12-12T02:25:24.463 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47170.log 2023-12-12T02:25:24.464 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63281.log 2023-12-12T02:25:24.464 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51275.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50868.log.gz 2023-12-12T02:25:24.464 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51275.log.gz 2023-12-12T02:25:24.464 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47170.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67760.log 2023-12-12T02:25:24.465 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77538.log 2023-12-12T02:25:24.465 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63281.log: /var/log/ceph/cluster2-client.mirror.67760.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63281.log.gz 2023-12-12T02:25:24.465 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.72473.log 2023-12-12T02:25:24.466 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77538.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67760.log.gz 2023-12-12T02:25:24.466 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77538.log.gz 2023-12-12T02:25:24.466 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75535.log 2023-12-12T02:25:24.467 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72793.log 2023-12-12T02:25:24.467 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.72473.log: /var/log/ceph/cluster1-client.mirror.75535.log: 86.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75535.log.gz 2023-12-12T02:25:24.467 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.72473.log.gz 2023-12-12T02:25:24.469 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54855.log 2023-12-12T02:25:24.469 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65059.log 2023-12-12T02:25:24.469 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.54855.log: -- replaced with /var/log/ceph/cluster2-client.mirror.72793.log.gz 2023-12-12T02:25:24.469 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47170.log.gz 2023-12-12T02:25:24.469 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60981.log 2023-12-12T02:25:24.470 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64746.log 2023-12-12T02:25:24.470 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65059.log: /var/log/ceph/cluster2-client.mirror.60981.log: 88.5% 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54855.log.gz 2023-12-12T02:25:24.470 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.60981.log.gz 2023-12-12T02:25:24.471 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65059.log.gz 2023-12-12T02:25:24.472 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52357.log 2023-12-12T02:25:24.472 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48164.log 2023-12-12T02:25:24.473 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52357.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50601.log 2023-12-12T02:25:24.473 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48164.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52357.log.gz 2023-12-12T02:25:24.473 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48164.log.gz 2023-12-12T02:25:24.473 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50187.log 2023-12-12T02:25:24.473 INFO:teuthology.orchestra.run.smithi140.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64746.log.gz 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71470.log 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68145.log 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50601.log.gz 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50187.log: /var/log/ceph/cluster2-client.mirror.71470.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43928.log 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50187.log.gz 2023-12-12T02:25:24.475 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68145.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.71470.log.gz 2023-12-12T02:25:24.476 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68145.log.gz 2023-12-12T02:25:24.476 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46148.log 2023-12-12T02:25:24.478 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.43928.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74805.log 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46148.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.43928.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54198.log 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74805.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46148.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.74805.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68310.log 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73236.log 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66122.log 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0%/var/log/ceph/cluster2-client.mirror.73236.log: -- replaced with /var/log/ceph/cluster2-client.mirror.54198.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68310.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73236.log.gz 2023-12-12T02:25:24.479 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44123.log 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58795.log 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66122.log: /var/log/ceph/cluster2-client.mirror.44123.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48559.log 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58795.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44123.log.gz 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66122.log.gz 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65507.log 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42768.log 2023-12-12T02:25:24.481 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48559.log: /var/log/ceph/cluster1-client.mirror.65507.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58795.log.gz 2023-12-12T02:25:24.483 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65507.log.gz 2023-12-12T02:25:24.483 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48559.log.gz 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55482.log 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42768.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54607.log 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67569.log 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.42768.log.gz 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55482.log.gz 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60335.log 2023-12-12T02:25:24.484 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76479.log 2023-12-12T02:25:24.486 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60335.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67569.log.gz 2023-12-12T02:25:24.486 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60335.log.gz 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79889.log 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61066.log 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54607.log.gz 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76479.log: /var/log/ceph/cluster2-client.admin.79889.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79889.log.gz 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76479.log.gz 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80045.log 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61066.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57988.log 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80045.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61066.log.gz 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47941.log 2023-12-12T02:25:24.487 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80045.log.gz 2023-12-12T02:25:24.489 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79603.log 2023-12-12T02:25:24.489 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57988.log: /var/log/ceph/cluster2-client.mirror.47941.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65217.log 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7%/var/log/ceph/cluster2-client.admin.79603.log: -- replaced with /var/log/ceph/cluster1-client.mirror.57988.log.gz 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79603.log.gz 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47941.log.gz 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69346.log 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68475.log 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65217.log: /var/log/ceph/cluster1-client.mirror.69346.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44661.log 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr: 85.2%/var/log/ceph/cluster2-client.mirror.68475.log: -- replaced with /var/log/ceph/cluster1-client.mirror.69346.log.gz 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.65217.log.gz 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53217.log 2023-12-12T02:25:24.490 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44386.log 2023-12-12T02:25:24.494 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44661.log: /var/log/ceph/cluster2-client.mirror.53217.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44661.log.gz 90.1% 2023-12-12T02:25:24.494 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.68475.log.gz 2023-12-12T02:25:24.494 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42133.log 2023-12-12T02:25:24.494 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44386.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56502.log 2023-12-12T02:25:24.494 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42133.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42133.log.gz 2023-12-12T02:25:24.499 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64788.log 2023-12-12T02:25:24.500 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56502.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50047.log.gz 2023-12-12T02:25:24.501 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56502.log.gz 2023-12-12T02:25:24.501 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50209.log 2023-12-12T02:25:24.502 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40446.log 2023-12-12T02:25:24.502 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64788.log: /var/log/ceph/cluster2-client.mirror.50209.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64788.log.gz 2023-12-12T02:25:24.503 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69886.log 2023-12-12T02:25:24.503 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.40446.log: 66.4% -- replaced with /var/log/ceph/cluster1-client.admin.40446.log.gz 2023-12-12T02:25:24.504 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60243.log 2023-12-12T02:25:24.504 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69886.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69886.log.gz 2023-12-12T02:25:24.504 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66825.log 2023-12-12T02:25:24.505 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60243.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.60243.log.gz 2023-12-12T02:25:24.506 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73277.log 2023-12-12T02:25:24.506 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66825.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66825.log.gz 2023-12-12T02:25:24.507 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61229.log 2023-12-12T02:25:24.507 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73277.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73277.log.gz 2023-12-12T02:25:24.508 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51841.log 2023-12-12T02:25:24.509 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61229.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.61229.log.gz 2023-12-12T02:25:24.509 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.78046.log 2023-12-12T02:25:24.509 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51841.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51841.log.gz 2023-12-12T02:25:24.510 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55053.log 2023-12-12T02:25:24.510 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.78046.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.78046.log.gz 2023-12-12T02:25:24.511 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68537.log 2023-12-12T02:25:24.512 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55053.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55053.log.gz 2023-12-12T02:25:24.512 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79265.log 2023-12-12T02:25:24.513 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68537.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68537.log.gz 2023-12-12T02:25:24.514 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68331.log 2023-12-12T02:25:24.514 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79265.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79265.log.gz 2023-12-12T02:25:24.515 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59473.log 2023-12-12T02:25:24.515 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68331.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.68331.log.gz 2023-12-12T02:25:24.516 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49234.log 2023-12-12T02:25:24.517 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59473.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59473.log.gz 2023-12-12T02:25:24.518 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79447.log 2023-12-12T02:25:24.518 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49234.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49234.log.gz 2023-12-12T02:25:24.519 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76290.log 2023-12-12T02:25:24.519 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79447.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79447.log.gz 2023-12-12T02:25:24.519 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39850.log 2023-12-12T02:25:24.520 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76290.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76290.log.gz 2023-12-12T02:25:24.520 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47981.log 2023-12-12T02:25:24.521 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.39850.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39850.log.gz 2023-12-12T02:25:24.521 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77859.log 2023-12-12T02:25:24.522 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47981.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47981.log.gz 2023-12-12T02:25:24.523 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68871.log 2023-12-12T02:25:24.523 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77859.log: 82.6% -- replaced with /var/log/ceph/cluster2-client.mirror.77859.log.gz 2023-12-12T02:25:24.523 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69391.log 2023-12-12T02:25:24.524 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68871.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68871.log.gz 2023-12-12T02:25:24.524 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45301.log 2023-12-12T02:25:24.525 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69391.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69391.log.gz 2023-12-12T02:25:24.525 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55505.log 2023-12-12T02:25:24.527 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45301.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53217.log.gz 2023-12-12T02:25:24.531 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45301.log.gz 2023-12-12T02:25:24.531 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42646.log 2023-12-12T02:25:24.531 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55505.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75512.log 2023-12-12T02:25:24.533 INFO:teuthology.orchestra.run.smithi140.stderr: 87.3%/var/log/ceph/cluster1-client.mirror.42646.log: -- replaced with /var/log/ceph/cluster1-client.mirror.55505.log.gz 2023-12-12T02:25:24.533 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79421.log 2023-12-12T02:25:24.533 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75512.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.75512.log.gz 2023-12-12T02:25:24.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42334.log 2023-12-12T02:25:24.534 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79421.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79421.log.gz 2023-12-12T02:25:24.534 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61356.log 2023-12-12T02:25:24.535 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.42334.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.42334.log.gz 2023-12-12T02:25:24.535 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51239.log 2023-12-12T02:25:24.536 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61356.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61356.log.gz 2023-12-12T02:25:24.536 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46796.log 2023-12-12T02:25:24.537 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51239.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42646.log.gz 2023-12-12T02:25:24.538 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51312.log 2023-12-12T02:25:24.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46796.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46796.log.gz 2023-12-12T02:25:24.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46171.log 2023-12-12T02:25:24.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51312.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51312.log.gz 2023-12-12T02:25:24.540 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51334.log 2023-12-12T02:25:24.541 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46171.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46171.log.gz 2023-12-12T02:25:24.542 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65571.log 2023-12-12T02:25:24.555 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51334.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43342.log 2023-12-12T02:25:24.557 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65571.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65571.log.gz 2023-12-12T02:25:24.571 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68768.log 2023-12-12T02:25:24.572 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43342.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43342.log.gz 2023-12-12T02:25:24.583 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50130.log.gz 2023-12-12T02:25:24.583 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75167.log 2023-12-12T02:25:24.584 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68768.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68768.log.gz 2023-12-12T02:25:24.584 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58933.log 2023-12-12T02:25:24.585 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75167.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75167.log.gz 2023-12-12T02:25:24.585 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56826.log 2023-12-12T02:25:24.586 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58933.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.58933.log.gz 2023-12-12T02:25:24.587 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70426.log 2023-12-12T02:25:24.587 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56826.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56826.log.gz 2023-12-12T02:25:24.588 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59580.log 2023-12-12T02:25:24.588 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70426.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70426.log.gz 2023-12-12T02:25:24.589 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66536.log 2023-12-12T02:25:24.589 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59580.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59580.log.gz 2023-12-12T02:25:24.590 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68682.log 2023-12-12T02:25:24.590 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66536.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66536.log.gz 2023-12-12T02:25:24.591 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53323.log 2023-12-12T02:25:24.592 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68682.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68682.log.gz 2023-12-12T02:25:24.592 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59649.log 2023-12-12T02:25:24.593 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53323.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53323.log.gz 2023-12-12T02:25:24.594 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.45895.log 2023-12-12T02:25:24.594 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59649.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59649.log.gz 2023-12-12T02:25:24.595 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62203.log 2023-12-12T02:25:24.607 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.2.45895.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45688.log 2023-12-12T02:25:24.608 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62203.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.62203.log.gz 2023-12-12T02:25:24.619 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69324.log 2023-12-12T02:25:24.630 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45688.log: 93.5% 92.9% -- replaced with /var/log/ceph/cluster1-client.mirror.3.40088.log.gz 2023-12-12T02:25:24.635 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75719.log 2023-12-12T02:25:24.636 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69324.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69324.log.gz 2023-12-12T02:25:24.636 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54008.log 2023-12-12T02:25:24.638 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75719.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75719.log.gz 2023-12-12T02:25:24.638 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53416.log 2023-12-12T02:25:24.638 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54008.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45688.log.gz 2023-12-12T02:25:24.647 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46548.log 2023-12-12T02:25:24.647 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53416.log: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53416.log.gz 2023-12-12T02:25:24.659 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.43388.log 2023-12-12T02:25:24.659 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46548.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46548.log.gz 2023-12-12T02:25:24.660 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49126.log 2023-12-12T02:25:24.660 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.43388.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.admin.43388.log.gz 2023-12-12T02:25:24.675 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56415.log 2023-12-12T02:25:24.675 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49126.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49126.log.gz 2023-12-12T02:25:24.688 INFO:teuthology.orchestra.run.smithi140.stderr: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70539.log 2023-12-12T02:25:24.689 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56415.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56415.log.gz 2023-12-12T02:25:24.690 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54008.log.gz 2023-12-12T02:25:24.690 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67114.log 2023-12-12T02:25:24.691 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70539.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70539.log.gz 2023-12-12T02:25:24.691 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41009.log 2023-12-12T02:25:24.693 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67114.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67114.log.gz 2023-12-12T02:25:24.693 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74953.log 2023-12-12T02:25:24.694 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41009.log: 57.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41009.log.gz 2023-12-12T02:25:24.694 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49859.log 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74953.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74953.log.gz 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60735.log 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49859.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49859.log.gz 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.45895.log.gz 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53437.log 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60735.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60735.log.gz 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66990.log 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44258.log 2023-12-12T02:25:24.700 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53437.log: /var/log/ceph/cluster2-client.mirror.66990.log: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53437.log.gz 2023-12-12T02:25:24.701 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72620.log 2023-12-12T02:25:24.701 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44258.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66990.log.gz 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44258.log.gz 2023-12-12T02:25:24.701 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.701 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44452.log 2023-12-12T02:25:24.702 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43846.log 2023-12-12T02:25:24.702 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72620.log: /var/log/ceph/cluster1-client.mirror.44452.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72620.log.gz 2023-12-12T02:25:24.702 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44452.log.gz 2023-12-12T02:25:24.702 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62734.log 2023-12-12T02:25:24.702 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74994.log 2023-12-12T02:25:24.703 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43846.log: /var/log/ceph/cluster2-client.mirror.62734.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62734.log.gz 2023-12-12T02:25:24.715 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41065.log 2023-12-12T02:25:24.716 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74994.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74994.log.gz 2023-12-12T02:25:24.716 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53785.log 2023-12-12T02:25:24.717 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41065.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41065.log.gz 2023-12-12T02:25:24.717 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71034.log 2023-12-12T02:25:24.718 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53785.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53785.log.gz 2023-12-12T02:25:24.718 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54877.log 2023-12-12T02:25:24.719 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71034.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.71034.log.gz 2023-12-12T02:25:24.719 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73963.log 2023-12-12T02:25:24.721 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54877.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54877.log.gz 2023-12-12T02:25:24.721 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57291.log 2023-12-12T02:25:24.722 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73963.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73963.log.gz 2023-12-12T02:25:24.722 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63815.log 2023-12-12T02:25:24.723 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57291.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57291.log.gz 2023-12-12T02:25:24.723 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47016.log 2023-12-12T02:25:24.724 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63815.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63815.log.gz 2023-12-12T02:25:24.724 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43558.log 2023-12-12T02:25:24.730 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47016.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47016.log.gz 2023-12-12T02:25:24.731 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80513.log 2023-12-12T02:25:24.732 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43558.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.43558.log.gz 2023-12-12T02:25:24.732 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58098.log 2023-12-12T02:25:24.732 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80513.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80513.log.gz 2023-12-12T02:25:24.733 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58297.log 2023-12-12T02:25:24.735 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58098.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.58098.log.gz 2023-12-12T02:25:24.735 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76414.log 2023-12-12T02:25:24.736 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58297.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.58297.log.gz 2023-12-12T02:25:24.737 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58685.log 2023-12-12T02:25:24.738 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76414.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.76414.log.gz 2023-12-12T02:25:24.738 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67404.log 2023-12-12T02:25:24.739 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58685.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58685.log.gz 2023-12-12T02:25:24.739 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68788.log 2023-12-12T02:25:24.740 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67404.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.67404.log.gz 2023-12-12T02:25:24.740 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55931.log 2023-12-12T02:25:24.742 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68788.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68788.log.gz 2023-12-12T02:25:24.742 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56981.log 2023-12-12T02:25:24.743 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55931.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55931.log.gz 2023-12-12T02:25:24.744 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39670.log 2023-12-12T02:25:24.745 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56981.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56981.log.gz 2023-12-12T02:25:24.745 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77836.log 2023-12-12T02:25:24.746 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39670.log.gz 2023-12-12T02:25:24.746 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77429.log 2023-12-12T02:25:24.747 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77836.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77836.log.gz 2023-12-12T02:25:24.747 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72912.log 2023-12-12T02:25:24.748 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77429.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77429.log.gz 2023-12-12T02:25:24.748 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47795.log 2023-12-12T02:25:24.749 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72912.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72912.log.gz 2023-12-12T02:25:24.750 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60018.log 2023-12-12T02:25:24.755 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47795.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47795.log.gz 2023-12-12T02:25:24.756 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76705.log 2023-12-12T02:25:24.756 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60018.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.60018.log.gz 2023-12-12T02:25:24.757 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65240.log 2023-12-12T02:25:24.758 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76705.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76705.log.gz 2023-12-12T02:25:24.758 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57554.log 2023-12-12T02:25:24.759 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65240.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65240.log.gz 2023-12-12T02:25:24.759 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51999.log 2023-12-12T02:25:24.760 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57554.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57554.log.gz 2023-12-12T02:25:24.760 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39975.log 2023-12-12T02:25:24.761 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51999.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51999.log.gz 2023-12-12T02:25:24.761 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67383.log 2023-12-12T02:25:24.762 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39975.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39975.log.gz 2023-12-12T02:25:24.763 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66639.log 2023-12-12T02:25:24.763 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67383.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67383.log.gz 2023-12-12T02:25:24.764 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64992.log 2023-12-12T02:25:24.765 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66639.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66639.log.gz 2023-12-12T02:25:24.765 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69549.log 2023-12-12T02:25:24.766 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64992.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64992.log.gz 2023-12-12T02:25:24.766 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63258.log 2023-12-12T02:25:24.767 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69549.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.69549.log.gz 2023-12-12T02:25:24.767 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57097.log 2023-12-12T02:25:24.768 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63258.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63258.log.gz 2023-12-12T02:25:24.769 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62537.log 2023-12-12T02:25:24.769 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57097.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57097.log.gz 2023-12-12T02:25:24.770 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69016.log 2023-12-12T02:25:24.771 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62537.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62537.log.gz 2023-12-12T02:25:24.771 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46431.log 2023-12-12T02:25:24.772 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.69016.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69016.log.gz 2023-12-12T02:25:24.772 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63594.log 2023-12-12T02:25:24.773 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46431.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46431.log.gz 2023-12-12T02:25:24.774 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59065.log 2023-12-12T02:25:24.775 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63594.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63594.log.gz 2023-12-12T02:25:24.775 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57531.log 2023-12-12T02:25:24.776 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59065.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59065.log.gz 2023-12-12T02:25:24.776 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78251.log 2023-12-12T02:25:24.777 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57531.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.57531.log.gz 2023-12-12T02:25:24.777 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45281.log 2023-12-12T02:25:24.778 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78251.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78251.log.gz 2023-12-12T02:25:24.778 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40079.log 2023-12-12T02:25:24.784 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45281.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45281.log.gz 2023-12-12T02:25:24.784 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73417.log 2023-12-12T02:25:24.785 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40079.log: 21.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40079.log.gz 2023-12-12T02:25:24.785 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75631.log 2023-12-12T02:25:24.786 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73417.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73417.log.gz 2023-12-12T02:25:24.787 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77494.log 2023-12-12T02:25:24.787 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75631.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75631.log.gz 2023-12-12T02:25:24.788 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57600.log 2023-12-12T02:25:24.789 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77494.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.77494.log.gz 2023-12-12T02:25:24.789 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62629.log 2023-12-12T02:25:24.790 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57600.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.57600.log.gz 2023-12-12T02:25:24.790 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44893.log 2023-12-12T02:25:24.791 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62629.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62629.log.gz 2023-12-12T02:25:24.791 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79083.log 2023-12-12T02:25:24.793 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44893.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44893.log.gz 2023-12-12T02:25:24.793 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59713.log 2023-12-12T02:25:24.794 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79083.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79083.log.gz 2023-12-12T02:25:24.794 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78303.log 2023-12-12T02:25:24.796 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59713.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59713.log.gz 2023-12-12T02:25:24.796 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69706.log 2023-12-12T02:25:24.797 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78303.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78303.log.gz 2023-12-12T02:25:24.797 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52821.log 2023-12-12T02:25:24.799 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69706.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69706.log.gz 2023-12-12T02:25:24.799 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2023-12-12T02:25:24.799 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76805.log 2023-12-12T02:25:24.811 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52821.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67722.log 2023-12-12T02:25:24.827 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76805.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48542.log 2023-12-12T02:25:24.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67722.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45366.log 2023-12-12T02:25:24.839 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48542.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48542.log.gz 2023-12-12T02:25:24.845 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51239.log.gz 2023-12-12T02:25:24.846 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80201.log 2023-12-12T02:25:24.847 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45366.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45366.log.gz 2023-12-12T02:25:24.848 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51180.log 2023-12-12T02:25:24.848 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80201.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80201.log.gz 2023-12-12T02:25:24.848 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66391.log 2023-12-12T02:25:24.849 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51180.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51180.log.gz 2023-12-12T02:25:24.849 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45731.log 2023-12-12T02:25:24.850 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66391.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66391.log.gz 2023-12-12T02:25:24.850 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73339.log 2023-12-12T02:25:24.856 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45731.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45731.log.gz 2023-12-12T02:25:24.856 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78745.log 2023-12-12T02:25:24.857 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73339.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73339.log.gz 2023-12-12T02:25:24.858 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46908.log 2023-12-12T02:25:24.858 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78745.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78745.log.gz 2023-12-12T02:25:24.858 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67631.log 2023-12-12T02:25:24.859 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46908.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46908.log.gz 2023-12-12T02:25:24.859 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.53217.log 2023-12-12T02:25:24.860 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67631.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67631.log.gz 2023-12-12T02:25:24.860 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65038.log 2023-12-12T02:25:24.875 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.53217.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53153.log 2023-12-12T02:25:24.876 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65038.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65038.log.gz 2023-12-12T02:25:24.883 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58057.log 2023-12-12T02:25:24.884 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53153.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53153.log.gz 2023-12-12T02:25:24.886 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43846.log.gz 2023-12-12T02:25:24.887 INFO:teuthology.orchestra.run.smithi140.stderr: 94.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67722.log.gz 2023-12-12T02:25:24.902 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49185.log 2023-12-12T02:25:24.902 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58057.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.58057.log.gz 2023-12-12T02:25:24.902 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58360.log 2023-12-12T02:25:24.908 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49185.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51334.log.gz 2023-12-12T02:25:24.909 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68227.log 2023-12-12T02:25:24.910 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58360.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58360.log.gz 2023-12-12T02:25:24.910 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59930.log 2023-12-12T02:25:24.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68227.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68227.log.gz 2023-12-12T02:25:24.912 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62846.log 2023-12-12T02:25:24.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59930.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.53217.log.gz 2023-12-12T02:25:24.914 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.59930.log.gz 2023-12-12T02:25:24.915 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50688.log 2023-12-12T02:25:24.915 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53937.log 2023-12-12T02:25:24.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62846.log: /var/log/ceph/cluster2-client.mirror.50688.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62846.log.gz 2023-12-12T02:25:24.916 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50688.log.gz 2023-12-12T02:25:24.916 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52298.log 2023-12-12T02:25:24.917 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51779.log 2023-12-12T02:25:24.917 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53937.log: /var/log/ceph/cluster2-client.mirror.52298.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53937.log.gz 2023-12-12T02:25:24.917 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52298.log.gz 2023-12-12T02:25:24.917 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40050.log 2023-12-12T02:25:24.919 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66019.log 2023-12-12T02:25:24.919 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51779.log: /var/log/ceph/cluster2-client.admin.40050.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40050.log.gz 2023-12-12T02:25:24.919 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51779.log.gz 2023-12-12T02:25:24.919 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54984.log 2023-12-12T02:25:24.920 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70741.log 2023-12-12T02:25:24.920 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66019.log: /var/log/ceph/cluster1-client.mirror.54984.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54984.log.gz 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66019.log.gz 2023-12-12T02:25:24.920 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.920 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44235.log 2023-12-12T02:25:24.921 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78823.log 2023-12-12T02:25:24.921 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70741.log: /var/log/ceph/cluster1-client.mirror.44235.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70741.log.gz 2023-12-12T02:25:24.921 INFO:teuthology.orchestra.run.smithi140.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44235.log.gz 2023-12-12T02:25:24.921 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64702.log 2023-12-12T02:25:24.922 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78823.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78823.log.gz 2023-12-12T02:25:24.922 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62244.log 2023-12-12T02:25:24.923 INFO:teuthology.orchestra.run.smithi140.stderr:gzip 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50209.log.gz -5 2023-12-12T02:25:24.923 INFO:teuthology.orchestra.run.smithi140.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.67590.log 2023-12-12T02:25:24.923 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64702.log: /var/log/ceph/cluster1-client.mirror.62244.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.64702.log.gz 2023-12-12T02:25:24.923 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47193.log 2023-12-12T02:25:24.924 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75958.log 2023-12-12T02:25:24.924 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67590.log: /var/log/ceph/cluster2-client.mirror.47193.log: 87.8% 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.62244.log.gz 2023-12-12T02:25:24.925 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67590.log.gz 2023-12-12T02:25:24.925 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74932.log 2023-12-12T02:25:24.925 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74309.log 2023-12-12T02:25:24.926 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75958.log: /var/log/ceph/cluster2-client.mirror.74932.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74932.log.gz 2023-12-12T02:25:24.926 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75958.log.gz 2023-12-12T02:25:24.926 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67507.log 2023-12-12T02:25:24.927 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39900.log 2023-12-12T02:25:24.927 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74309.log: /var/log/ceph/cluster2-client.mirror.67507.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.74309.log.gz 2023-12-12T02:25:24.928 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.42714.log 2023-12-12T02:25:24.928 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67507.log.gz 2023-12-12T02:25:24.929 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.39900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39900.log.gz 2023-12-12T02:25:24.929 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63413.log 2023-12-12T02:25:24.929 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72538.log 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.42714.log: /var/log/ceph/cluster2-client.mirror.63413.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47193.log.gz 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.42714.log.gz 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63413.log.gz 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61983.log 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40083.log 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72538.log: /var/log/ceph/cluster2-client.mirror.61983.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55402.log 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40083.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72538.log.gz 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr: 80.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40083.log.gz 2023-12-12T02:25:24.931 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.45986.log 2023-12-12T02:25:24.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72452.log 2023-12-12T02:25:24.932 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55402.log: /var/log/ceph/cluster2-client.mirror.3.45986.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.61983.log.gz 2023-12-12T02:25:24.932 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76500.log 2023-12-12T02:25:24.933 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72452.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.72452.log.gz 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55402.log.gz 2023-12-12T02:25:24.933 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.934 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51641.log 2023-12-12T02:25:24.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68269.log 2023-12-12T02:25:24.935 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76500.log: /var/log/ceph/cluster2-client.mirror.51641.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76500.log.gz 2023-12-12T02:25:24.935 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51641.log.gz 2023-12-12T02:25:24.935 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69036.log 2023-12-12T02:25:24.936 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68269.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68269.log.gz 2023-12-12T02:25:24.936 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61714.log 2023-12-12T02:25:24.936 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39443.log 2023-12-12T02:25:24.937 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.69036.log: /var/log/ceph/cluster1-client.mirror.61714.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61714.log.gz 2023-12-12T02:25:24.937 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69036.log.gz 2023-12-12T02:25:24.938 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63391.log 2023-12-12T02:25:24.938 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70404.log 2023-12-12T02:25:24.939 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39443.log: /var/log/ceph/cluster1-client.mirror.63391.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39443.log.gz 2023-12-12T02:25:24.939 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52174.log 2023-12-12T02:25:24.939 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70404.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70404.log.gz 2023-12-12T02:25:24.940 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52957.log 2023-12-12T02:25:24.940 INFO:teuthology.orchestra.run.smithi140.stderr: 91.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63391.log.gz 2023-12-12T02:25:24.941 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52174.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52174.log.gz 2023-12-12T02:25:24.941 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63982.log 2023-12-12T02:25:24.942 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80279.log 2023-12-12T02:25:24.942 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52957.log: /var/log/ceph/cluster1-client.mirror.63982.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52957.log.gz 2023-12-12T02:25:24.943 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63982.log.gz 2023-12-12T02:25:24.943 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39774.log 2023-12-12T02:25:24.944 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47256.log 2023-12-12T02:25:24.944 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80279.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80279.log.gz 2023-12-12T02:25:24.944 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39774.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49669.log 2023-12-12T02:25:24.944 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.39774.log.gz 2023-12-12T02:25:24.949 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47256.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47256.log.gz 2023-12-12T02:25:24.949 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54961.log 2023-12-12T02:25:24.950 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.64355.log 2023-12-12T02:25:24.950 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49669.log: /var/log/ceph/cluster1-client.mirror.54961.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54961.log.gz 88.8% 2023-12-12T02:25:24.950 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49669.log.gz 2023-12-12T02:25:24.951 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73840.log 2023-12-12T02:25:24.951 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50767.log 2023-12-12T02:25:24.952 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.64355.log: /var/log/ceph/cluster2-client.mirror.73840.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73840.log.gz 2023-12-12T02:25:24.956 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44386.log.gz 2023-12-12T02:25:24.963 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60579.log 2023-12-12T02:25:24.963 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53739.log 2023-12-12T02:25:24.964 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50767.log: /var/log/ceph/cluster1-client.mirror.60579.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50767.log.gz 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60579.log.gz 2023-12-12T02:25:24.964 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.964 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65957.log 2023-12-12T02:25:24.965 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53739.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78875.log 2023-12-12T02:25:24.965 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.65957.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53739.log.gz 2023-12-12T02:25:24.965 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68579.log 2023-12-12T02:25:24.965 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65957.log.gz 2023-12-12T02:25:24.967 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40204.log 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78875.log: /var/log/ceph/cluster2-client.mirror.68579.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78875.log.gz 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.64355.log.gz 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67238.log 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68579.log.gz 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74606.log 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40204.log: /var/log/ceph/cluster2-client.mirror.67238.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63548.log 2023-12-12T02:25:24.968 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74606.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40204.log.gz 2023-12-12T02:25:24.969 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51700.log 2023-12-12T02:25:24.969 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.74606.log.gz 2023-12-12T02:25:24.969 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63548.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.67238.log.gz 2023-12-12T02:25:24.969 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63548.log.gz 2023-12-12T02:25:24.969 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73318.log 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51700.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56244.log 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67694.log 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51700.log.gz 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56244.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.73318.log.gz 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43512.log 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56244.log.gz 2023-12-12T02:25:24.971 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54310.log 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43512.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60712.log 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67694.log.gz 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54310.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43512.log.gz 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64321.log 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.60712.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54310.log.gz 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55224.log 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60712.log.gz 2023-12-12T02:25:24.973 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64321.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64321.log.gz 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68207.log 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55224.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51118.log 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68207.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55224.log.gz 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75286.log 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67342.log 2023-12-12T02:25:24.975 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.51118.log: -- replaced with /var/log/ceph/cluster2-client.mirror.68207.log.gz 2023-12-12T02:25:24.976 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75286.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75286.log.gz 2023-12-12T02:25:24.976 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51118.log.gz 2023-12-12T02:25:24.976 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79005.log 2023-12-12T02:25:24.977 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67342.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55116.log 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79005.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74685.log 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.67342.log.gz 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79005.log.gz 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39332.log 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50905.log 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55116.log.gz 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74685.log: /var/log/ceph/cluster2-client.admin.39332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39332.log.gz 2023-12-12T02:25:24.978 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74685.log.gz 2023-12-12T02:25:24.981 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64355.log 2023-12-12T02:25:24.981 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70156.log 2023-12-12T02:25:24.981 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66970.log 2023-12-12T02:25:24.981 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50905.log.gz 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70156.log: 94.7%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.64355.log.gz -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46324.log 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70156.log.gz 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52821.log.gz 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66970.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58887.log 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66970.log.gz 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44725.log 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58887.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60379.log 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.77643.log 2023-12-12T02:25:24.982 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58887.log.gz 2023-12-12T02:25:24.983 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60379.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.60379.log.gz 2023-12-12T02:25:24.983 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52727.log 2023-12-12T02:25:24.985 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.77643.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.77643.log.gz 2023-12-12T02:25:24.986 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42668.log 2023-12-12T02:25:24.986 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46324.log.gz 2023-12-12T02:25:24.987 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53983.log 2023-12-12T02:25:24.987 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39410.log 2023-12-12T02:25:24.987 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44725.log.gz 2023-12-12T02:25:24.987 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42668.log: /var/log/ceph/cluster1-client.mirror.53983.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.42668.log.gz 2023-12-12T02:25:24.987 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53983.log.gz 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71453.log 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78953.log 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39410.log.gz 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.71453.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43733.log 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71453.log.gz 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47838.log 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78953.log: /var/log/ceph/cluster1-client.mirror.43733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.41201.log 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78953.log.gz 2023-12-12T02:25:24.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47838.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69999.log 2023-12-12T02:25:24.994 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.41201.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43733.log.gz 2023-12-12T02:25:24.994 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47838.log.gz 2023-12-12T02:25:24.994 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63901.log 2023-12-12T02:25:24.994 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69999.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40000.log 2023-12-12T02:25:24.995 INFO:teuthology.orchestra.run.smithi140.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69999.log.gz 2023-12-12T02:25:24.995 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63901.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64512.log 2023-12-12T02:25:24.995 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.40000.log: 88.4% 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.63901.log.gz 2023-12-12T02:25:24.995 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.40000.log.gz 2023-12-12T02:25:24.996 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68892.log 2023-12-12T02:25:24.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69504.log 2023-12-12T02:25:24.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68892.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64512.log.gz 2023-12-12T02:25:24.997 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68892.log.gz 2023-12-12T02:25:24.998 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48376.log 2023-12-12T02:25:24.998 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62180.log 2023-12-12T02:25:24.998 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69504.log: /var/log/ceph/cluster2-client.mirror.48376.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69504.log.gz 2023-12-12T02:25:24.998 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48376.log.gz 2023-12-12T02:25:24.998 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49839.log 2023-12-12T02:25:24.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70629.log 2023-12-12T02:25:24.999 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62180.log: /var/log/ceph/cluster2-client.mirror.49839.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62180.log.gz 2023-12-12T02:25:24.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79811.log 2023-12-12T02:25:24.999 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49839.log.gz 2023-12-12T02:25:25.000 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70629.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70629.log.gz 2023-12-12T02:25:25.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63703.log 2023-12-12T02:25:25.001 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.60806.log 2023-12-12T02:25:25.001 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79811.log: /var/log/ceph/cluster2-client.mirror.63703.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79811.log.gz 2023-12-12T02:25:25.002 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45006.log 2023-12-12T02:25:25.002 INFO:teuthology.orchestra.run.smithi140.stderr: 88.9%/var/log/ceph/cluster1-client.mirror.0.60806.log: -- replaced with /var/log/ceph/cluster2-client.mirror.63703.log.gz 2023-12-12T02:25:25.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77153.log 2023-12-12T02:25:25.005 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45006.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45006.log.gz 2023-12-12T02:25:25.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63177.log 2023-12-12T02:25:25.006 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77153.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77153.log.gz 2023-12-12T02:25:25.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48943.log 2023-12-12T02:25:25.008 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63177.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63177.log.gz 2023-12-12T02:25:25.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79395.log 2023-12-12T02:25:25.009 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48943.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48943.log.gz 2023-12-12T02:25:25.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63525.log 2023-12-12T02:25:25.010 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79395.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79395.log.gz 2023-12-12T02:25:25.010 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75249.log 2023-12-12T02:25:25.011 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63525.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63525.log.gz 2023-12-12T02:25:25.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68455.log 2023-12-12T02:25:25.012 INFO:teuthology.orchestra.run.smithi140.stderr: 94.3%/var/log/ceph/cluster2-client.mirror.75249.log: -- replaced with /var/log/ceph/cluster1-client.mirror.0.41201.log.gz 2023-12-12T02:25:25.012 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75249.log.gz 2023-12-12T02:25:25.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78096.log 2023-12-12T02:25:25.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60289.log 2023-12-12T02:25:25.013 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68455.log: /var/log/ceph/cluster2-client.mirror.78096.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.78096.log.gz 2023-12-12T02:25:25.013 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.68455.log.gz 2023-12-12T02:25:25.013 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39592.log 2023-12-12T02:25:25.014 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60692.log 2023-12-12T02:25:25.014 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60289.log: /var/log/ceph/cluster2-client.admin.39592.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39592.log.gz 2023-12-12T02:25:25.014 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60289.log.gz 2023-12-12T02:25:25.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69436.log 2023-12-12T02:25:25.015 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80019.log 2023-12-12T02:25:25.016 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60692.log: /var/log/ceph/cluster1-client.mirror.69436.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69436.log.gz 2023-12-12T02:25:25.016 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60692.log.gz 2023-12-12T02:25:25.016 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67073.log 2023-12-12T02:25:25.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44322.log 2023-12-12T02:25:25.017 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80019.log: /var/log/ceph/cluster2-client.mirror.67073.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80019.log.gz 2023-12-12T02:25:25.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80409.log 2023-12-12T02:25:25.018 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67073.log.gz 2023-12-12T02:25:25.023 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44322.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44322.log.gz 2023-12-12T02:25:25.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67176.log 2023-12-12T02:25:25.023 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80409.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46862.log 2023-12-12T02:25:25.023 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80409.log.gz 2023-12-12T02:25:25.024 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67176.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58979.log 2023-12-12T02:25:25.024 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46862.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46862.log.gz 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64458.log 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.67176.log.gz/var/log/ceph/cluster1-client.mirror.58979.log: 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50167.log 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.58979.log.gz 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64458.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67011.log 2023-12-12T02:25:25.026 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50167.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64458.log.gz 2023-12-12T02:25:25.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79161.log 2023-12-12T02:25:25.027 INFO:teuthology.orchestra.run.smithi140.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50167.log.gz 2023-12-12T02:25:25.028 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41937.log 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79161.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.67011.log.gz 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79161.log.gz 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48514.log 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42539.log 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.41937.log: /var/log/ceph/cluster2-client.mirror.48514.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.41937.log.gz 2023-12-12T02:25:25.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48734.log 2023-12-12T02:25:25.030 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42539.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42539.log.gz 2023-12-12T02:25:25.031 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40178.log 2023-12-12T02:25:25.031 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48734.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48734.log.gz 2023-12-12T02:25:25.032 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75699.log 2023-12-12T02:25:25.032 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40178.log: 77.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40178.log.gz 2023-12-12T02:25:25.032 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58418.log 2023-12-12T02:25:25.033 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75699.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75699.log.gz 2023-12-12T02:25:25.033 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74782.log 2023-12-12T02:25:25.034 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58418.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.58418.log.gz 2023-12-12T02:25:25.035 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53016.log 2023-12-12T02:25:25.035 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74782.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.74782.log.gz 2023-12-12T02:25:25.036 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50394.log 2023-12-12T02:25:25.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53016.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53016.log.gz 2023-12-12T02:25:25.037 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61143.log 2023-12-12T02:25:25.037 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50394.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50394.log.gz 2023-12-12T02:25:25.038 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.40083.log 2023-12-12T02:25:25.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61143.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61143.log.gz 2023-12-12T02:25:25.039 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39748.log 2023-12-12T02:25:25.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.40083.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.40083.log.gz 2023-12-12T02:25:25.047 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51858.log 2023-12-12T02:25:25.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39748.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39748.log.gz 2023-12-12T02:25:25.048 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47036.log 2023-12-12T02:25:25.048 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51858.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51858.log.gz 2023-12-12T02:25:25.049 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49143.log 2023-12-12T02:25:25.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47036.log: 93.7% -- replaced with /var/log/ceph/cluster2-client.mirror.3.45986.log.gz 2023-12-12T02:25:25.050 INFO:teuthology.orchestra.run.smithi140.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.60806.log.gz 2023-12-12T02:25:25.054 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47036.log.gz 2023-12-12T02:25:25.054 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80773.log 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61207.log 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59278.log 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% 88.8% -- replaced with /var/log/ceph/cluster2-client.admin.80773.log.gz 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49143.log.gz 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61207.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42447.log 2023-12-12T02:25:25.056 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62133.log 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59278.log: /var/log/ceph/cluster2-client.mirror.42447.log: 88.4% 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42447.log.gz 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59278.log.gz 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56000.log 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61207.log.gz 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62133.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52878.log 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56000.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62133.log.gz 2023-12-12T02:25:25.058 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73820.log 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.60223.log 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52878.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56000.log.gz 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73820.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52878.log.gz 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73820.log.gz 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56265.log 2023-12-12T02:25:25.060 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.60223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77071.log 2023-12-12T02:25:25.061 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56265.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52648.log 2023-12-12T02:25:25.061 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.60223.log.gz 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76541.log 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.77071.log.gz 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52648.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43101.log 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56265.log.gz/var/log/ceph/cluster2-client.mirror.76541.log: 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76541.log.gz 2023-12-12T02:25:25.062 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75308.log 2023-12-12T02:25:25.063 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76249.log 2023-12-12T02:25:25.063 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43101.log: /var/log/ceph/cluster2-client.mirror.75308.log: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.43101.log.gz 2023-12-12T02:25:25.064 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48184.log 2023-12-12T02:25:25.064 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76249.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76249.log.gz 2023-12-12T02:25:25.065 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65101.log 2023-12-12T02:25:25.065 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48184.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48184.log.gz 2023-12-12T02:25:25.066 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76349.log 2023-12-12T02:25:25.066 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65101.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65101.log.gz 2023-12-12T02:25:25.067 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76520.log 2023-12-12T02:25:25.083 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52421.log 2023-12-12T02:25:25.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76520.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76520.log.gz 2023-12-12T02:25:25.091 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76370.log 2023-12-12T02:25:25.092 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.52421.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52421.log.gz 2023-12-12T02:25:25.119 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57383.log 2023-12-12T02:25:25.119 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76370.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.76370.log.gz 2023-12-12T02:25:25.135 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46346.log 2023-12-12T02:25:25.136 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57383.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.57383.log.gz 2023-12-12T02:25:25.147 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77174.log 2023-12-12T02:25:25.148 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46346.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46346.log.gz 2023-12-12T02:25:25.159 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78511.log 2023-12-12T02:25:25.159 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77174.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77174.log.gz 2023-12-12T02:25:25.171 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54835.log 2023-12-12T02:25:25.171 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78511.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78511.log.gz 2023-12-12T02:25:25.183 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53570.log 2023-12-12T02:25:25.185 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54835.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54835.log.gz 2023-12-12T02:25:25.190 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48514.log.gz 2023-12-12T02:25:25.190 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55074.log 2023-12-12T02:25:25.191 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53570.log: 3.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53570.log.gz 2023-12-12T02:25:25.191 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52510.log 2023-12-12T02:25:25.194 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55074.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55074.log.gz 2023-12-12T02:25:25.194 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72681.log 2023-12-12T02:25:25.195 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52510.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52510.log.gz 2023-12-12T02:25:25.195 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42156.log 2023-12-12T02:25:25.196 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72681.log: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.72681.log.gz 2023-12-12T02:25:25.196 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52216.log 2023-12-12T02:25:25.196 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42156.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42156.log.gz 2023-12-12T02:25:25.197 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69078.log 2023-12-12T02:25:25.211 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52216.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51663.log 2023-12-12T02:25:25.211 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.69078.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.69078.log.gz 2023-12-12T02:25:25.227 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50564.log 2023-12-12T02:25:25.242 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51663.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52727.log.gz 2023-12-12T02:25:25.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.50475.log 2023-12-12T02:25:25.243 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50564.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50564.log.gz 2023-12-12T02:25:25.243 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.40079.log 2023-12-12T02:25:25.249 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.50475.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.75308.log.gz 2023-12-12T02:25:25.259 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64533.log 2023-12-12T02:25:25.259 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.0.40079.log: 78.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.40079.log.gz 2023-12-12T02:25:25.265 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76805.log.gz 2023-12-12T02:25:25.266 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52920.log 2023-12-12T02:25:25.268 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64533.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64533.log.gz 2023-12-12T02:25:25.268 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63155.log 2023-12-12T02:25:25.269 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52920.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52920.log.gz 2023-12-12T02:25:25.269 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68124.log 2023-12-12T02:25:25.271 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63155.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63155.log.gz 2023-12-12T02:25:25.272 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50411.log 2023-12-12T02:25:25.272 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68124.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68124.log.gz 2023-12-12T02:25:25.273 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57701.log 2023-12-12T02:25:25.274 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50411.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50411.log.gz 2023-12-12T02:25:25.274 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49314.log 2023-12-12T02:25:25.277 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57701.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57701.log.gz 2023-12-12T02:25:25.277 INFO:teuthology.orchestra.run.smithi140.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.50475.log.gz 2023-12-12T02:25:25.277 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70269.log 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49314.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59603.log 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49314.log.gz/var/log/ceph/cluster1-client.mirror.70269.log: 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70269.log.gz 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76085.log 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59603.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40958.log 2023-12-12T02:25:25.278 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76085.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59603.log.gz 2023-12-12T02:25:25.282 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44212.log 2023-12-12T02:25:25.282 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.76085.log.gz 2023-12-12T02:25:25.282 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40958.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62670.log 2023-12-12T02:25:25.282 INFO:teuthology.orchestra.run.smithi140.stderr: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40958.log.gz 2023-12-12T02:25:25.282 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44212.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40480.log 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5%/var/log/ceph/cluster1-client.mirror.62670.log: -- replaced with /var/log/ceph/cluster1-client.mirror.44212.log.gz 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61777.log 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40480.log: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40480.log.gz 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78459.log 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4%/var/log/ceph/cluster2-client.mirror.61777.log: -- replaced with /var/log/ceph/cluster1-client.mirror.62670.log.gz 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63879.log 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.61777.log.gz 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78459.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46661.log 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78459.log.gz 2023-12-12T02:25:25.283 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63879.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54520.log 2023-12-12T02:25:25.284 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46661.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.63879.log.gz 2023-12-12T02:25:25.284 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51356.log 2023-12-12T02:25:25.285 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54520.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54520.log.gz 2023-12-12T02:25:25.285 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40497.log 2023-12-12T02:25:25.286 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.51356.log: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51356.log.gz 2023-12-12T02:25:25.288 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56959.log 2023-12-12T02:25:25.288 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40497.log: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40497.log.gz 2023-12-12T02:25:25.288 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46661.log.gz 2023-12-12T02:25:25.288 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45191.log 2023-12-12T02:25:25.288 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56959.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70494.log 2023-12-12T02:25:25.289 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45191.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45191.log.gz 2023-12-12T02:25:25.290 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46724.log 2023-12-12T02:25:25.290 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56959.log.gz 2023-12-12T02:25:25.290 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70494.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70494.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57922.log 2023-12-12T02:25:25.290 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:25.291 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73359.log 2023-12-12T02:25:25.292 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57922.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.57922.log.gz 2023-12-12T02:25:25.292 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68372.log 2023-12-12T02:25:25.294 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73359.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.73359.log.gz 2023-12-12T02:25:25.294 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76105.log 2023-12-12T02:25:25.295 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68372.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68372.log.gz 2023-12-12T02:25:25.296 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65874.log 2023-12-12T02:25:25.296 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46724.log.gz 2023-12-12T02:25:25.296 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76105.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76105.log.gz 2023-12-12T02:25:25.296 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66143.log 2023-12-12T02:25:25.297 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62479.log 2023-12-12T02:25:25.297 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65874.log: /var/log/ceph/cluster2-client.mirror.66143.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66143.log.gz 2023-12-12T02:25:25.298 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65874.log.gz 2023-12-12T02:25:25.299 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49067.log 2023-12-12T02:25:25.299 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71236.log 2023-12-12T02:25:25.300 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62479.log: /var/log/ceph/cluster2-client.mirror.49067.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.62479.log.gz 2023-12-12T02:25:25.300 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49067.log.gz 2023-12-12T02:25:25.300 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56461.log 2023-12-12T02:25:25.301 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45388.log 2023-12-12T02:25:25.301 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71236.log: /var/log/ceph/cluster1-client.mirror.56461.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.71236.log.gz 2023-12-12T02:25:25.301 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56461.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48061.log 2023-12-12T02:25:25.301 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:25.315 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45388.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54722.log 2023-12-12T02:25:25.316 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48061.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48061.log.gz 2023-12-12T02:25:25.316 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65916.log 2023-12-12T02:25:25.317 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54722.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54722.log.gz 2023-12-12T02:25:25.317 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75466.log 2023-12-12T02:25:25.318 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65916.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65916.log.gz 2023-12-12T02:25:25.319 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50005.log 2023-12-12T02:25:25.320 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75466.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.75466.log.gz 2023-12-12T02:25:25.320 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49163.log 2023-12-12T02:25:25.320 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49185.log.gz 2023-12-12T02:25:25.321 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80877.log 2023-12-12T02:25:25.321 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49163.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50005.log.gz 2023-12-12T02:25:25.322 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49163.log.gz 2023-12-12T02:25:25.323 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65895.log 2023-12-12T02:25:25.323 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.40178.log 2023-12-12T02:25:25.323 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80877.log: /var/log/ceph/cluster2-client.mirror.65895.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80877.log.gz 2023-12-12T02:25:25.323 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51762.log 2023-12-12T02:25:25.323 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65895.log.gz 2023-12-12T02:25:25.324 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.40178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.67910.log 2023-12-12T02:25:25.324 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51762.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51762.log.gz 2023-12-12T02:25:25.324 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52157.log 2023-12-12T02:25:25.326 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.67910.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.1.40178.log.gz 2023-12-12T02:25:25.327 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51138.log 2023-12-12T02:25:25.327 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52157.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52157.log.gz 2023-12-12T02:25:25.328 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62823.log 2023-12-12T02:25:25.328 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51138.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51138.log.gz 2023-12-12T02:25:25.329 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46950.log 2023-12-12T02:25:25.330 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62823.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62823.log.gz 2023-12-12T02:25:25.330 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47236.log 2023-12-12T02:25:25.331 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46950.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.67910.log.gz 2023-12-12T02:25:25.332 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77194.log 2023-12-12T02:25:25.336 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47236.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46950.log.gz 2023-12-12T02:25:25.336 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48435.log 2023-12-12T02:25:25.338 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77194.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77194.log.gz 2023-12-12T02:25:25.338 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47236.log.gz 2023-12-12T02:25:25.338 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.71435.log 2023-12-12T02:25:25.338 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76684.log 2023-12-12T02:25:25.339 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.71435.log: 3.9% -- replaced with /var/log/ceph/cluster2-client.mirror.71435.log.gz 2023-12-12T02:25:25.339 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68933.log 2023-12-12T02:25:25.339 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55595.log 2023-12-12T02:25:25.340 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34978.log 2023-12-12T02:25:25.341 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72640.log 2023-12-12T02:25:25.341 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66763.log 2023-12-12T02:25:25.341 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75014.log 2023-12-12T02:25:25.341 INFO:teuthology.orchestra.run.smithi140.stderr: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76684.log.gz 2023-12-12T02:25:25.342 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63350.log 2023-12-12T02:25:25.342 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64575.log 2023-12-12T02:25:25.342 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56718.log 2023-12-12T02:25:25.343 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66763.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64613.log 2023-12-12T02:25:25.343 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56718.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66763.log.gz 2023-12-12T02:25:25.344 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68933.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68933.log.gz 2023-12-12T02:25:25.344 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56718.log.gz 2023-12-12T02:25:25.345 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70089.log 2023-12-12T02:25:25.346 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64613.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.64613.log.gz 2023-12-12T02:25:25.346 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51937.log 2023-12-12T02:25:25.347 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70089.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70089.log.gz 2023-12-12T02:25:25.348 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74095.log 2023-12-12T02:25:25.348 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51937.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51937.log.gz 2023-12-12T02:25:25.349 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47365.log 2023-12-12T02:25:25.350 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74095.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.74095.log.gz 2023-12-12T02:25:25.351 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80591.log 2023-12-12T02:25:25.355 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47365.log: /var/log/ceph/cluster2-client.mirror.55595.log: /var/log/ceph/cluster2-client.admin.34978.log: /var/log/ceph/cluster2-client.mirror.72640.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34978.log.gz 2023-12-12T02:25:25.355 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75014.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.72640.log.gz 2023-12-12T02:25:25.355 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63350.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75014.log.gz 2023-12-12T02:25:25.356 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63350.log.gz 2023-12-12T02:25:25.356 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55595.log.gz 2023-12-12T02:25:25.357 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47365.log.gz 2023-12-12T02:25:25.357 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74286.log 2023-12-12T02:25:25.358 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80591.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80591.log.gz 2023-12-12T02:25:25.358 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65674.log 2023-12-12T02:25:25.359 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.74286.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74286.log.gz 2023-12-12T02:25:25.359 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43364.log 2023-12-12T02:25:25.360 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65674.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65674.log.gz 2023-12-12T02:25:25.361 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72704.log 2023-12-12T02:25:25.371 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43364.log: /var/log/ceph/cluster2-client.mirror.64575.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64575.log.gz 2023-12-12T02:25:25.373 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52648.log.gz 2023-12-12T02:25:25.375 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63135.log 2023-12-12T02:25:25.376 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72704.log: 85.3% -- replaced with /var/log/ceph/cluster2-client.mirror.72704.log.gz 2023-12-12T02:25:25.376 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61023.log 2023-12-12T02:25:25.378 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63135.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63135.log.gz 2023-12-12T02:25:25.378 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77092.log 2023-12-12T02:25:25.380 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61023.log: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.61023.log.gz 2023-12-12T02:25:25.380 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46525.log 2023-12-12T02:25:25.381 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77092.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77092.log.gz 2023-12-12T02:25:25.381 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55424.log 2023-12-12T02:25:25.382 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46525.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46525.log.gz 2023-12-12T02:25:25.383 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52841.log 2023-12-12T02:25:25.383 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55424.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55424.log.gz 2023-12-12T02:25:25.384 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58510.log 2023-12-12T02:25:25.384 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52841.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52841.log.gz 2023-12-12T02:25:25.385 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50623.log 2023-12-12T02:25:25.386 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58510.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.58510.log.gz 2023-12-12T02:25:25.386 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67362.log 2023-12-12T02:25:25.403 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50623.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67259.log 2023-12-12T02:25:25.405 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67362.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.67362.log.gz 2023-12-12T02:25:25.405 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51663.log.gz 2023-12-12T02:25:25.411 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48021.log 2023-12-12T02:25:25.412 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67259.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67259.log.gz 2023-12-12T02:25:25.412 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.77009.log 2023-12-12T02:25:25.413 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48021.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.48021.log.gz 2023-12-12T02:25:25.413 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45552.log 2023-12-12T02:25:25.414 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.77009.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.77009.log.gz 2023-12-12T02:25:25.414 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53762.log 2023-12-12T02:25:25.415 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.45552.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45552.log.gz 2023-12-12T02:25:25.416 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.67197.log 2023-12-12T02:25:25.417 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.53762.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.53762.log.gz 2023-12-12T02:25:25.417 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65327.log 2023-12-12T02:25:25.418 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.67197.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.67197.log.gz 2023-12-12T02:25:25.418 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68912.log 2023-12-12T02:25:25.421 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65327.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65327.log.gz 2023-12-12T02:25:25.421 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73257.log 2023-12-12T02:25:25.423 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68912.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68912.log.gz 2023-12-12T02:25:25.423 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40429.log 2023-12-12T02:25:25.424 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73257.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73257.log.gz 2023-12-12T02:25:25.424 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44702.log 2023-12-12T02:25:25.425 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.40429.log: 80.0% -- replaced with /var/log/ceph/cluster1-client.admin.40429.log.gz 2023-12-12T02:25:25.425 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61488.log 2023-12-12T02:25:25.431 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44702.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44702.log.gz 2023-12-12T02:25:25.431 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69864.log 2023-12-12T02:25:25.432 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61488.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.61488.log.gz 2023-12-12T02:25:25.433 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55831.log 2023-12-12T02:25:25.433 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69864.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.69864.log.gz 2023-12-12T02:25:25.434 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73380.log 2023-12-12T02:25:25.436 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55831.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55831.log.gz 2023-12-12T02:25:25.436 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66102.log 2023-12-12T02:25:25.437 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73380.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.73380.log.gz 2023-12-12T02:25:25.437 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68517.log 2023-12-12T02:25:25.438 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66102.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66102.log.gz 2023-12-12T02:25:25.439 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63021.log 2023-12-12T02:25:25.440 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68517.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68517.log.gz 2023-12-12T02:25:25.440 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55615.log 2023-12-12T02:25:25.441 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63021.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.63021.log.gz 2023-12-12T02:25:25.441 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50088.log 2023-12-12T02:25:25.443 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55615.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55615.log.gz 2023-12-12T02:25:25.444 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43121.log 2023-12-12T02:25:25.445 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50088.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50088.log.gz 2023-12-12T02:25:25.445 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41308.log 2023-12-12T02:25:25.446 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43121.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.43121.log.gz 2023-12-12T02:25:25.447 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59386.log 2023-12-12T02:25:25.459 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.41308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68599.log 2023-12-12T02:25:25.460 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59386.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59386.log.gz 2023-12-12T02:25:25.475 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74441.log 2023-12-12T02:25:25.477 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68599.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.68599.log.gz 2023-12-12T02:25:25.477 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47921.log 2023-12-12T02:25:25.478 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74441.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74441.log.gz 2023-12-12T02:25:25.491 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77235.log 2023-12-12T02:25:25.492 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47921.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47921.log.gz 2023-12-12T02:25:25.498 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52216.log.gz 2023-12-12T02:25:25.498 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73986.log 2023-12-12T02:25:25.499 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77235.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.77235.log.gz 2023-12-12T02:25:25.499 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62692.log 2023-12-12T02:25:25.500 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73986.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.73986.log.gz 2023-12-12T02:25:25.500 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69684.log 2023-12-12T02:25:25.501 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62692.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62692.log.gz 2023-12-12T02:25:25.501 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.76458.log 2023-12-12T02:25:25.502 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69684.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69684.log.gz 2023-12-12T02:25:25.502 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51504.log 2023-12-12T02:25:25.503 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.76458.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.76458.log.gz 2023-12-12T02:25:25.503 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50789.log 2023-12-12T02:25:25.515 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51504.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75800.log 2023-12-12T02:25:25.539 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80149.log 2023-12-12T02:25:25.540 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75800.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76349.log.gz 2023-12-12T02:25:25.543 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51604.log 2023-12-12T02:25:25.543 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80149.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80149.log.gz 2023-12-12T02:25:25.544 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69098.log 2023-12-12T02:25:25.545 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51604.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51604.log.gz 2023-12-12T02:25:25.545 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41855.log 2023-12-12T02:25:25.547 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.69098.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.69098.log.gz 2023-12-12T02:25:25.547 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62393.log 2023-12-12T02:25:25.548 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.41855.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.admin.41855.log.gz 2023-12-12T02:25:25.548 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.54008.log 2023-12-12T02:25:25.549 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62393.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62393.log.gz 2023-12-12T02:25:25.549 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56023.log 2023-12-12T02:25:25.563 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.1.54008.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45388.log.gz 2023-12-12T02:25:25.563 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74586.log 2023-12-12T02:25:25.564 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56023.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56023.log.gz 2023-12-12T02:25:25.564 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55178.log 2023-12-12T02:25:25.565 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74586.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74586.log.gz 2023-12-12T02:25:25.565 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77051.log 2023-12-12T02:25:25.566 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55178.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55178.log.gz 2023-12-12T02:25:25.566 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52320.log 2023-12-12T02:25:25.567 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77051.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77051.log.gz 2023-12-12T02:25:25.568 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59693.log 2023-12-12T02:25:25.579 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63091.log 2023-12-12T02:25:25.581 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59693.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59693.log.gz 2023-12-12T02:25:25.591 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70651.log 2023-12-12T02:25:25.592 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63091.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63091.log.gz 2023-12-12T02:25:25.592 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63959.log 2023-12-12T02:25:25.593 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70651.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.70651.log.gz 2023-12-12T02:25:25.607 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43144.log 2023-12-12T02:25:25.608 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63959.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63959.log.gz 2023-12-12T02:25:25.608 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48413.log 2023-12-12T02:25:25.609 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43144.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43144.log.gz 2023-12-12T02:25:25.609 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77775.log 2023-12-12T02:25:25.610 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48413.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48413.log.gz 2023-12-12T02:25:25.620 INFO:teuthology.orchestra.run.smithi140.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.1.54008.log.gz 2023-12-12T02:25:25.620 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76307.log 2023-12-12T02:25:25.621 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77775.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77775.log.gz 2023-12-12T02:25:25.622 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55249.log 2023-12-12T02:25:25.623 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76307.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.76307.log.gz 2023-12-12T02:25:25.623 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54160.log 2023-12-12T02:25:25.625 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55249.log: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55249.log.gz 2023-12-12T02:25:25.625 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77473.log 2023-12-12T02:25:25.626 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54160.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54160.log.gz 2023-12-12T02:25:25.627 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.75078.log 2023-12-12T02:25:25.627 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77473.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.77473.log.gz 2023-12-12T02:25:25.628 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76044.log 2023-12-12T02:25:25.630 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.75078.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.75078.log.gz 2023-12-12T02:25:25.630 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69145.log 2023-12-12T02:25:25.631 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76044.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76044.log.gz 2023-12-12T02:25:25.632 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70786.log 2023-12-12T02:25:25.632 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69145.log: 5.2% -- replaced with /var/log/ceph/cluster1-client.mirror.69145.log.gz 2023-12-12T02:25:25.633 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65697.log 2023-12-12T02:25:25.634 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70786.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70786.log.gz 2023-12-12T02:25:25.634 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78355.log 2023-12-12T02:25:25.635 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65697.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65697.log.gz 2023-12-12T02:25:25.635 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64115.log 2023-12-12T02:25:25.636 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78355.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78355.log.gz 2023-12-12T02:25:25.636 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34758.log 2023-12-12T02:25:25.638 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64115.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64115.log.gz 2023-12-12T02:25:25.638 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72934.log 2023-12-12T02:25:25.639 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.34758.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34758.log.gz 2023-12-12T02:25:25.639 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45575.log 2023-12-12T02:25:25.646 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72934.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48435.log.gz 2023-12-12T02:25:25.655 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63327.log 2023-12-12T02:25:25.655 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45575.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45575.log.gz 2023-12-12T02:25:25.656 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61893.log 2023-12-12T02:25:25.657 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63327.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63327.log.gz 2023-12-12T02:25:25.675 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64725.log 2023-12-12T02:25:25.675 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61893.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61893.log.gz 2023-12-12T02:25:25.683 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44789.log 2023-12-12T02:25:25.683 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64725.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.64725.log.gz 2023-12-12T02:25:25.699 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68620.log 2023-12-12T02:25:25.711 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53806.log 2023-12-12T02:25:25.712 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68620.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68620.log.gz 2023-12-12T02:25:25.719 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31083.log 2023-12-12T02:25:25.720 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53806.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53806.log.gz 2023-12-12T02:25:25.735 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47537.log 2023-12-12T02:25:25.735 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/ceph.tmp-client.admin.31083.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31083.log.gz 2023-12-12T02:25:25.751 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75757.log 2023-12-12T02:25:25.759 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47537.log: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68703.log 2023-12-12T02:25:25.760 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75757.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.75757.log.gz 2023-12-12T02:25:25.770 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47537.log.gz 2023-12-12T02:25:25.774 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50623.log.gz 2023-12-12T02:25:25.775 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74705.log 2023-12-12T02:25:25.775 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68703.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.68703.log.gz 2023-12-12T02:25:25.795 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66329.log 2023-12-12T02:25:25.796 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74705.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74705.log.gz 2023-12-12T02:25:25.796 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43364.log.gz 2023-12-12T02:25:25.797 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39281.log 2023-12-12T02:25:25.797 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66329.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66329.log.gz 2023-12-12T02:25:25.804 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63615.log 2023-12-12T02:25:25.811 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39281.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39281.log.gz 2023-12-12T02:25:25.819 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50789.log.gz 2023-12-12T02:25:25.823 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47858.log 2023-12-12T02:25:25.825 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63615.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63615.log.gz 2023-12-12T02:25:25.825 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52399.log 2023-12-12T02:25:25.828 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47858.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52320.log.gz 2023-12-12T02:25:25.831 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47858.log.gz 2023-12-12T02:25:25.831 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55810.log 2023-12-12T02:25:25.832 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54264.log 2023-12-12T02:25:25.833 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52399.log: /var/log/ceph/cluster1-client.mirror.55810.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55810.log.gz 2023-12-12T02:25:25.839 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56221.log 2023-12-12T02:25:25.840 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54264.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54264.log.gz 2023-12-12T02:25:25.840 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77215.log 2023-12-12T02:25:25.841 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56221.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.56221.log.gz 2023-12-12T02:25:25.841 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44682.log 2023-12-12T02:25:25.842 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77215.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.77215.log.gz 2023-12-12T02:25:25.842 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79473.log 2023-12-12T02:25:25.848 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44682.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44682.log.gz 2023-12-12T02:25:25.848 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74891.log 2023-12-12T02:25:25.849 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79473.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79473.log.gz 2023-12-12T02:25:25.849 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56152.log 2023-12-12T02:25:25.850 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74891.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.74891.log.gz 2023-12-12T02:25:25.851 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40548.log 2023-12-12T02:25:25.852 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56152.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56152.log.gz 2023-12-12T02:25:25.852 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.78190.log 2023-12-12T02:25:25.853 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40548.log: 54.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40548.log.gz 2023-12-12T02:25:25.853 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60895.log 2023-12-12T02:25:25.854 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.78190.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.78190.log.gz 2023-12-12T02:25:25.854 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42110.log 2023-12-12T02:25:25.855 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60895.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60895.log.gz 2023-12-12T02:25:25.855 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75266.log 2023-12-12T02:25:25.856 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42110.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42110.log.gz 2023-12-12T02:25:25.856 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44592.log 2023-12-12T02:25:25.858 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75266.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.75266.log.gz 2023-12-12T02:25:25.858 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73693.log 2023-12-12T02:25:25.859 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44592.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44592.log.gz 2023-12-12T02:25:25.859 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64095.log 2023-12-12T02:25:25.860 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.73693.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.73693.log.gz 2023-12-12T02:25:25.860 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39522.log 2023-12-12T02:25:25.863 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64095.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64095.log.gz 2023-12-12T02:25:25.863 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49351.log 2023-12-12T02:25:25.864 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39522.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39522.log.gz 2023-12-12T02:25:25.864 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58487.log 2023-12-12T02:25:25.865 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49351.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49351.log.gz 2023-12-12T02:25:25.865 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75330.log 2023-12-12T02:25:25.866 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58487.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.58487.log.gz 2023-12-12T02:25:25.866 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42583.log 2023-12-12T02:25:25.867 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.75330.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.75330.log.gz 2023-12-12T02:25:25.867 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69616.log 2023-12-12T02:25:25.873 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42583.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42583.log.gz 2023-12-12T02:25:25.873 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62800.log 2023-12-12T02:25:25.874 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69616.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69616.log.gz 2023-12-12T02:25:25.874 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52016.log 2023-12-12T02:25:25.875 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62800.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.62800.log.gz 2023-12-12T02:25:25.876 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.61287.log 2023-12-12T02:25:25.877 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52016.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52016.log.gz 2023-12-12T02:25:25.877 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58120.log 2023-12-12T02:25:25.878 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.61287.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.61287.log.gz 2023-12-12T02:25:25.878 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40582.log 2023-12-12T02:25:25.879 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58120.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58120.log.gz 2023-12-12T02:25:25.880 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80461.log 2023-12-12T02:25:25.880 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.40582.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.40582.log.gz 2023-12-12T02:25:25.881 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65484.log 2023-12-12T02:25:25.881 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80461.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80461.log.gz 2023-12-12T02:25:25.882 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42793.log 2023-12-12T02:25:25.883 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65484.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65484.log.gz 2023-12-12T02:25:25.883 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48306.log 2023-12-12T02:25:25.884 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42793.log: 81.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42793.log.gz 2023-12-12T02:25:25.884 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52569.log 2023-12-12T02:25:25.886 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48306.log: 90.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48306.log.gz 2023-12-12T02:25:25.886 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62517.log 2023-12-12T02:25:25.899 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52569.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44299.log 2023-12-12T02:25:25.900 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62517.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.62517.log.gz 2023-12-12T02:25:25.904 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41308.log.gz 2023-12-12T02:25:25.911 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61112.log 2023-12-12T02:25:25.911 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52194.log 2023-12-12T02:25:25.912 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44299.log: /var/log/ceph/cluster1-client.mirror.61112.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61112.log.gz 2023-12-12T02:25:25.912 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42690.log 2023-12-12T02:25:25.913 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52194.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52194.log.gz 2023-12-12T02:25:25.914 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51292.log 2023-12-12T02:25:25.916 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.42690.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44299.log.gz 2023-12-12T02:25:25.927 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66887.log 2023-12-12T02:25:25.928 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51292.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51292.log.gz 2023-12-12T02:25:25.943 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34827.log 2023-12-12T02:25:25.944 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66887.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66887.log.gz 2023-12-12T02:25:25.944 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51504.log.gz 2023-12-12T02:25:25.945 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45457.log 2023-12-12T02:25:25.945 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.34827.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34827.log.gz 2023-12-12T02:25:25.946 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69931.log 2023-12-12T02:25:25.947 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45457.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45457.log.gz 2023-12-12T02:25:25.947 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64679.log 2023-12-12T02:25:25.948 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69931.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69931.log.gz 2023-12-12T02:25:25.948 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42266.log 2023-12-12T02:25:25.949 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64679.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64679.log.gz 2023-12-12T02:25:25.950 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.69057.log 2023-12-12T02:25:25.951 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42266.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44789.log.gz 2023-12-12T02:25:25.955 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42266.log.gz 2023-12-12T02:25:25.955 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54918.log 2023-12-12T02:25:25.956 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43802.log 2023-12-12T02:25:25.957 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.69057.log: /var/log/ceph/cluster2-client.mirror.54918.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.69057.log.gz 2023-12-12T02:25:25.957 INFO:teuthology.orchestra.run.smithi140.stderr: 89.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54918.log.gz 2023-12-12T02:25:25.957 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66040.log 2023-12-12T02:25:25.958 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76146.log 2023-12-12T02:25:25.958 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66040.log: /var/log/ceph/cluster1-client.mirror.43802.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66040.log.gz 2023-12-12T02:25:25.962 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43802.log.gz 2023-12-12T02:25:25.963 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57229.log 2023-12-12T02:25:25.963 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.40634.log 2023-12-12T02:25:25.964 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76146.log: /var/log/ceph/cluster2-client.mirror.57229.log: 87.1% -- replaced with /var/log/ceph/cluster2-client.mirror.76146.log.gz 2023-12-12T02:25:25.964 INFO:teuthology.orchestra.run.smithi140.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57229.log.gz 2023-12-12T02:25:25.964 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58441.log 2023-12-12T02:25:25.965 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78641.log 2023-12-12T02:25:25.965 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.40634.log: /var/log/ceph/cluster1-client.mirror.58441.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.admin.40634.log.gz 2023-12-12T02:25:25.966 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58441.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78214.log 2023-12-12T02:25:25.966 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:25.966 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78641.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78641.log.gz 2023-12-12T02:25:25.967 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74332.log 2023-12-12T02:25:25.967 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61755.log 2023-12-12T02:25:25.967 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78214.log: /var/log/ceph/cluster1-client.mirror.74332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78214.log.gz 2023-12-12T02:25:25.967 INFO:teuthology.orchestra.run.smithi140.stderr: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.74332.log.gz 2023-12-12T02:25:25.967 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75937.log 2023-12-12T02:25:25.969 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61755.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.61755.log.gz 2023-12-12T02:25:25.969 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47279.log 2023-12-12T02:25:25.970 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55030.log 2023-12-12T02:25:25.970 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47279.log: /var/log/ceph/cluster1-client.mirror.75937.log: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.75937.log.gz 2023-12-12T02:25:25.971 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50453.log 2023-12-12T02:25:25.972 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55030.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55030.log.gz 2023-12-12T02:25:25.972 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.39925.log 2023-12-12T02:25:25.974 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50453.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.75800.log.gz 2023-12-12T02:25:25.974 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47279.log.gz 2023-12-12T02:25:25.975 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.72934.log.gz 2023-12-12T02:25:25.987 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.41829.log 2023-12-12T02:25:25.987 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.39925.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69256.log 2023-12-12T02:25:25.987 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.39925.log.gz 2023-12-12T02:25:25.987 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.41829.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47428.log 2023-12-12T02:25:25.989 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.69256.log: -- replaced with /var/log/ceph/cluster1-client.admin.41829.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58531.log 2023-12-12T02:25:25.989 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:25:25.989 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.69256.log.gz 2023-12-12T02:25:25.989 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47428.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.39678.log 2023-12-12T02:25:25.990 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.58531.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.73508.log 2023-12-12T02:25:25.990 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.39678.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.39678.log.gz 2023-12-12T02:25:25.990 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58531.log.gz 2023-12-12T02:25:25.991 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57337.log 2023-12-12T02:25:25.991 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.73508.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65998.log 2023-12-12T02:25:25.992 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57337.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57337.log.gz 2023-12-12T02:25:25.993 INFO:teuthology.orchestra.run.smithi140.stderr: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.0.73508.log.gz 2023-12-12T02:25:25.993 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56849.log 2023-12-12T02:25:25.994 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65998.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64969.log 2023-12-12T02:25:25.994 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47428.log.gz 2023-12-12T02:25:25.994 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56849.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56849.log.gz 2023-12-12T02:25:25.995 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.65998.log.gz 2023-12-12T02:25:25.995 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80123.log 2023-12-12T02:25:25.995 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64969.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46240.log 2023-12-12T02:25:25.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80123.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64969.log.gz 2023-12-12T02:25:25.996 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40086.log 2023-12-12T02:25:25.996 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80123.log.gz 2023-12-12T02:25:25.996 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.46240.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46240.log.gz 2023-12-12T02:25:25.997 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48980.log 2023-12-12T02:25:25.997 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50253.log 2023-12-12T02:25:25.997 INFO:teuthology.orchestra.run.smithi140.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.mirror.40086.log.gz 2023-12-12T02:25:25.998 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48980.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.75228.log 2023-12-12T02:25:25.998 INFO:teuthology.orchestra.run.smithi140.stderr:gzip 87.5% -5 -- replaced with /var/log/ceph/cluster2-client.mirror.48980.log.gz --verbose 2023-12-12T02:25:25.998 INFO:teuthology.orchestra.run.smithi140.stderr: -- /var/log/ceph/cluster1-client.mirror.42179.log 2023-12-12T02:25:25.999 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50253.log: /var/log/ceph/cluster2-client.mirror.75228.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50253.log.gz 2023-12-12T02:25:25.999 INFO:teuthology.orchestra.run.smithi140.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.75228.log.gz 2023-12-12T02:25:25.999 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79707.log 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48081.log 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42179.log: /var/log/ceph/cluster2-client.admin.79707.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47881.log 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42179.log.gz 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79707.log.gz 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.48081.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48081.log.gz 2023-12-12T02:25:26.000 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.58464.log 2023-12-12T02:25:26.001 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64160.log 2023-12-12T02:25:26.001 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.58464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54474.log 2023-12-12T02:25:26.002 INFO:teuthology.orchestra.run.smithi140.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47881.log.gz 2023-12-12T02:25:26.002 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.64160.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.58464.log.gz 2023-12-12T02:25:26.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50352.log 2023-12-12T02:25:26.003 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.44422.log 2023-12-12T02:25:26.004 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54474.log: /var/log/ceph/cluster2-client.mirror.50352.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54474.log.gz 2023-12-12T02:25:26.004 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71214.log 2023-12-12T02:25:26.004 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.50352.log.gz 2023-12-12T02:25:26.005 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.44422.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.44422.log.gz 2023-12-12T02:25:26.005 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66370.log 2023-12-12T02:25:26.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63502.log 2023-12-12T02:25:26.006 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.71214.log: /var/log/ceph/cluster2-client.mirror.66370.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71214.log.gz 2023-12-12T02:25:26.006 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60312.log 2023-12-12T02:25:26.007 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63502.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63502.log.gz 2023-12-12T02:25:26.007 INFO:teuthology.orchestra.run.smithi140.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66370.log.gz 2023-12-12T02:25:26.007 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80799.log 2023-12-12T02:25:26.008 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60312.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.60312.log.gz 2023-12-12T02:25:26.008 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40412.log 2023-12-12T02:25:26.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57051.log 2023-12-12T02:25:26.009 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80799.log: /var/log/ceph/cluster1-client.admin.40412.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80799.log.gz 2023-12-12T02:25:26.009 INFO:teuthology.orchestra.run.smithi140.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.40412.log.gz 2023-12-12T02:25:26.009 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60358.log 2023-12-12T02:25:26.010 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57051.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.57051.log.gz 2023-12-12T02:25:26.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69594.log 2023-12-12T02:25:26.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47299.log 2023-12-12T02:25:26.011 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60358.log: /var/log/ceph/cluster1-client.mirror.69594.log: 86.9% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60358.log.gz 2023-12-12T02:25:26.011 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.69594.log.gz 2023-12-12T02:25:26.011 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74263.log 2023-12-12T02:25:26.012 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59866.log 2023-12-12T02:25:26.012 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.47299.log: /var/log/ceph/cluster1-client.mirror.74263.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.74263.log.gz 2023-12-12T02:25:26.017 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47299.log.gz 2023-12-12T02:25:26.017 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2023-12-12T02:25:26.018 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62003.log 2023-12-12T02:25:26.019 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59866.log: /var/log/ceph/cluster2-osd.0.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59866.log.gz 2023-12-12T02:25:26.019 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76746.log 2023-12-12T02:25:26.021 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.62003.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.62003.log.gz 2023-12-12T02:25:26.021 INFO:teuthology.orchestra.run.smithi140.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.64160.log.gz 2023-12-12T02:25:26.022 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50651.log 2023-12-12T02:25:26.022 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76746.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46704.log 2023-12-12T02:25:26.022 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76746.log.gz 2023-12-12T02:25:26.022 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50651.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50651.log.gz 2023-12-12T02:25:26.023 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74376.log 2023-12-12T02:25:26.023 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46704.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42562.log 2023-12-12T02:25:26.023 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74376.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.74376.log.gz 2023-12-12T02:25:26.024 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69796.log 2023-12-12T02:25:26.024 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.42562.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42562.log.gz 2023-12-12T02:25:26.025 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43604.log 2023-12-12T02:25:26.025 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69796.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.69796.log.gz 2023-12-12T02:25:26.026 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76208.log 2023-12-12T02:25:26.026 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43604.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43604.log.gz 2023-12-12T02:25:26.027 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.34851.log 2023-12-12T02:25:26.027 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76208.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76208.log.gz 2023-12-12T02:25:26.028 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68723.log 2023-12-12T02:25:26.028 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46704.log.gz/var/log/ceph/cluster2-client.admin.34851.log: 2023-12-12T02:25:26.028 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.34851.log.gz 2023-12-12T02:25:26.029 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66866.log 2023-12-12T02:25:26.029 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77133.log 2023-12-12T02:25:26.030 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66866.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.68723.log.gz 2023-12-12T02:25:26.031 INFO:teuthology.orchestra.run.smithi140.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66866.log.gz 2023-12-12T02:25:26.031 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66205.log 2023-12-12T02:25:26.031 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.77133.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55741.log 2023-12-12T02:25:26.032 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66205.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.77133.log.gz 2023-12-12T02:25:26.032 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66205.log.gz 2023-12-12T02:25:26.033 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42288.log 2023-12-12T02:25:26.033 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.55741.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51799.log 2023-12-12T02:25:26.033 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.42288.log: -- replaced with /var/log/ceph/cluster1-client.mirror.55741.log.gz 2023-12-12T02:25:26.034 INFO:teuthology.orchestra.run.smithi140.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42288.log.gz 2023-12-12T02:25:26.034 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70021.log 2023-12-12T02:25:26.034 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51799.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80097.log 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70021.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51799.log.gz 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70021.log.gz 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76126.log 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44143.log 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80097.log.gz 2023-12-12T02:25:26.036 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76126.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.76126.log.gz 2023-12-12T02:25:26.037 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63792.log 2023-12-12T02:25:26.037 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.44143.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56285.log 2023-12-12T02:25:26.037 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63792.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44143.log.gz 2023-12-12T02:25:26.037 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63792.log.gz 2023-12-12T02:25:26.038 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76023.log 2023-12-12T02:25:26.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56285.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50431.log 2023-12-12T02:25:26.038 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.76023.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.76023.log.gz 2023-12-12T02:25:26.039 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65461.log 2023-12-12T02:25:26.040 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50431.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.56285.log.gz 2023-12-12T02:25:26.040 INFO:teuthology.orchestra.run.smithi140.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.50431.log.gz 2023-12-12T02:25:26.040 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.73175.log 2023-12-12T02:25:26.040 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57876.log 2023-12-12T02:25:26.041 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4%/var/log/ceph/cluster2-client.mirror.73175.log: -- replaced with /var/log/ceph/cluster1-client.mirror.65461.log.gz 2023-12-12T02:25:26.041 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.73175.log.gz 2023-12-12T02:25:26.042 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56198.log 2023-12-12T02:25:26.042 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.57876.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63235.log 2023-12-12T02:25:26.043 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56198.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57876.log.gz 2023-12-12T02:25:26.043 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56198.log.gz 2023-12-12T02:25:26.043 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55695.log 2023-12-12T02:25:26.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.63235.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56132.log 2023-12-12T02:25:26.044 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.55695.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63235.log.gz 2023-12-12T02:25:26.044 INFO:teuthology.orchestra.run.smithi140.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55695.log.gz 2023-12-12T02:25:26.044 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52569.log.gz 2023-12-12T02:25:26.044 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66784.log 2023-12-12T02:25:26.045 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.56132.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.75657.log 2023-12-12T02:25:26.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66784.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64028.log 2023-12-12T02:25:26.046 INFO:teuthology.orchestra.run.smithi140.stderr: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56132.log.gz 2023-12-12T02:25:26.046 INFO:teuthology.orchestra.run.smithi140.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66784.log.gz 2023-12-12T02:25:26.046 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.75657.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.75657.log.gz 2023-12-12T02:25:26.046 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.71101.log 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.69819.log 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5%/var/log/ceph/cluster1-client.mirror.71101.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64028.log.gz 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79187.log 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.71101.log.gz 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.69819.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.69819.log.gz 2023-12-12T02:25:26.047 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79863.log 2023-12-12T02:25:26.048 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.68744.log 2023-12-12T02:25:26.048 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79187.log.gz 2023-12-12T02:25:26.049 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79863.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78615.log 2023-12-12T02:25:26.049 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79863.log.gz 2023-12-12T02:25:26.049 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.68744.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.74420.log 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45751.log 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78615.log: /var/log/ceph/cluster1-client.mirror.74420.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.68744.log.gz 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78615.log.gz 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.44915.log 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.74420.log.gz 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.45751.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.53459.log 2023-12-12T02:25:26.050 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52799.log 2023-12-12T02:25:26.054 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.2.44915.log: /var/log/ceph/cluster2-client.mirror.1.53459.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45751.log.gz 2023-12-12T02:25:26.059 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49711.log 2023-12-12T02:25:26.059 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52799.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52799.log.gz 2023-12-12T02:25:26.060 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79681.log 2023-12-12T02:25:26.071 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49711.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65171.log 2023-12-12T02:25:26.071 INFO:teuthology.orchestra.run.smithi140.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42690.log.gz 2023-12-12T02:25:26.072 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79681.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79681.log.gz 2023-12-12T02:25:26.080 INFO:teuthology.orchestra.run.smithi140.stderr: 93.2% -- replaced with /var/log/ceph/cluster2-client.mirror.2.44915.log.gz 2023-12-12T02:25:26.083 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53306.log 2023-12-12T02:25:26.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53173.log 2023-12-12T02:25:26.084 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65171.log.gz 2023-12-12T02:25:26.084 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53306.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53306.log.gz 2023-12-12T02:25:26.084 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.76269.log 2023-12-12T02:25:26.085 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60151.log 2023-12-12T02:25:26.085 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53173.log: /var/log/ceph/cluster2-client.mirror.76269.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53173.log.gz 2023-12-12T02:25:26.085 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74524.log 2023-12-12T02:25:26.086 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60151.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.76269.log.gz 2023-12-12T02:25:26.088 INFO:teuthology.orchestra.run.smithi140.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.60151.log.gz 2023-12-12T02:25:26.088 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52399.log.gz 2023-12-12T02:25:26.088 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.78071.log 2023-12-12T02:25:26.089 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47580.log 2023-12-12T02:25:26.089 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74524.log: /var/log/ceph/cluster2-client.mirror.78071.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52527.log 2023-12-12T02:25:26.089 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.78071.log.gz 2023-12-12T02:25:26.089 INFO:teuthology.orchestra.run.smithi140.stderr: 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.74524.log.gz 2023-12-12T02:25:26.090 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.47580.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.53459.log.gz 2023-12-12T02:25:26.091 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.75823.log 2023-12-12T02:25:26.091 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52527.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49689.log 2023-12-12T02:25:26.092 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.0.75823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78537.log 2023-12-12T02:25:26.092 INFO:teuthology.orchestra.run.smithi140.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52527.log.gz 2023-12-12T02:25:26.092 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49689.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64946.log 2023-12-12T02:25:26.092 INFO:teuthology.orchestra.run.smithi140.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49689.log.gz 2023-12-12T02:25:26.093 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78537.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78537.log.gz 2023-12-12T02:25:26.093 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79785.log 2023-12-12T02:25:26.094 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.64946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50710.log 2023-12-12T02:25:26.094 INFO:teuthology.orchestra.run.smithi140.stderr: 87.4%/var/log/ceph/cluster2-client.admin.79785.log: -- replaced with /var/log/ceph/cluster1-client.mirror.64946.log.gz 2023-12-12T02:25:26.094 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79785.log.gz 2023-12-12T02:25:26.095 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47580.log.gz 2023-12-12T02:25:26.095 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40197.log 2023-12-12T02:25:26.095 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.50710.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.40364.log 2023-12-12T02:25:26.095 INFO:teuthology.orchestra.run.smithi140.stderr: 93.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.75823.log.gz 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.51356.log 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40197.log.gz 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.admin.40364.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62650.log 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.40364.log.gz 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.1.51356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80487.log 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43442.log 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.62650.log: /var/log/ceph/cluster2-client.admin.80487.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80487.log.gz 2023-12-12T02:25:26.097 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49536.log 2023-12-12T02:25:26.098 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.43442.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.62650.log.gz 2023-12-12T02:25:26.098 INFO:teuthology.orchestra.run.smithi140.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43442.log.gz 2023-12-12T02:25:26.099 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66846.log 2023-12-12T02:25:26.099 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57120.log 2023-12-12T02:25:26.099 INFO:teuthology.orchestra.run.smithi140.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49536.log.gz 2023-12-12T02:25:26.099 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66846.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54699.log 2023-12-12T02:25:26.100 INFO:teuthology.orchestra.run.smithi140.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66846.log.gz/var/log/ceph/cluster1-client.mirror.57120.log: 2023-12-12T02:25:26.100 INFO:teuthology.orchestra.run.smithi140.stderr: 87.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56064.log 2023-12-12T02:25:26.100 INFO:teuthology.orchestra.run.smithi140.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.57120.log.gz 2023-12-12T02:25:26.100 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54699.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70944.log 2023-12-12T02:25:26.101 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56064.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54699.log.gz 2023-12-12T02:25:26.101 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65807.log 2023-12-12T02:25:26.102 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70944.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70944.log.gz 2023-12-12T02:25:26.102 INFO:teuthology.orchestra.run.smithi140.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56064.log.gz 2023-12-12T02:25:26.102 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.77256.log 2023-12-12T02:25:26.103 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51742.log 2023-12-12T02:25:26.103 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.65807.log: /var/log/ceph/cluster2-client.mirror.77256.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.77256.log.gz 2023-12-12T02:25:26.104 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49553.log 2023-12-12T02:25:26.104 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.51742.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.65807.log.gz 2023-12-12T02:25:26.105 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40531.log 2023-12-12T02:25:26.106 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.49553.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49553.log.gz 2023-12-12T02:25:26.106 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70201.log 2023-12-12T02:25:26.106 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.40531.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40531.log.gz 2023-12-12T02:25:26.107 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80721.log 2023-12-12T02:25:26.107 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70201.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.70201.log.gz 2023-12-12T02:25:26.108 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2023-12-12T02:25:26.108 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80721.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80721.log.gz 2023-12-12T02:25:26.109 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.70134.log 2023-12-12T02:25:26.114 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-osd.2.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.51356.log.gz 2023-12-12T02:25:26.115 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54566.log 2023-12-12T02:25:26.115 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.70134.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.70134.log.gz 2023-12-12T02:25:26.116 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65787.log 2023-12-12T02:25:26.117 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.54566.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54566.log.gz 2023-12-12T02:25:26.117 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.80071.log 2023-12-12T02:25:26.119 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.65787.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65787.log.gz 2023-12-12T02:25:26.119 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59495.log 2023-12-12T02:25:26.120 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.80071.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.80071.log.gz 2023-12-12T02:25:26.120 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40992.log 2023-12-12T02:25:26.121 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.59495.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59495.log.gz 2023-12-12T02:25:26.122 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46304.log 2023-12-12T02:25:26.122 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.40992.log: 58.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40992.log.gz 2023-12-12T02:25:26.123 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61333.log 2023-12-12T02:25:26.130 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.46304.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46304.log.gz 2023-12-12T02:25:26.130 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56628.log 2023-12-12T02:25:26.130 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.61333.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.61333.log.gz 2023-12-12T02:25:26.130 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.57577.log 2023-12-12T02:25:26.131 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.56628.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56628.log.gz 2023-12-12T02:25:26.131 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59002.log 2023-12-12T02:25:26.132 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.57577.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.57577.log.gz 2023-12-12T02:25:26.132 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66908.log 2023-12-12T02:25:26.133 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.59002.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59002.log.gz 2023-12-12T02:25:26.134 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.72579.log 2023-12-12T02:25:26.135 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66908.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.66908.log.gz 2023-12-12T02:25:26.135 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44189.log 2023-12-12T02:25:26.136 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.72579.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.72579.log.gz 2023-12-12T02:25:26.136 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66742.log 2023-12-12T02:25:26.137 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.44189.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44189.log.gz 2023-12-12T02:25:26.137 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60758.log 2023-12-12T02:25:26.139 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.66742.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66742.log.gz 2023-12-12T02:25:26.139 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78719.log 2023-12-12T02:25:26.140 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.60758.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60758.log.gz 2023-12-12T02:25:26.141 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63067.log 2023-12-12T02:25:26.141 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78719.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78719.log.gz 2023-12-12T02:25:26.142 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52036.log 2023-12-12T02:25:26.142 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.63067.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63067.log.gz 2023-12-12T02:25:26.143 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.78667.log 2023-12-12T02:25:26.144 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.52036.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52036.log.gz 2023-12-12T02:25:26.144 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.74133.log 2023-12-12T02:25:26.145 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.78667.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.78667.log.gz 2023-12-12T02:25:26.145 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.79941.log 2023-12-12T02:25:26.146 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.74133.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.74133.log.gz 2023-12-12T02:25:26.146 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53195.log 2023-12-12T02:25:26.147 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.admin.79941.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.79941.log.gz 2023-12-12T02:25:26.147 INFO:teuthology.orchestra.run.smithi140.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43670.log 2023-12-12T02:25:26.163 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.53195.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54679.log 2023-12-12T02:25:26.164 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster1-client.mirror.43670.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43670.log.gz 2023-12-12T02:25:26.180 INFO:teuthology.orchestra.run.smithi140.stderr:/var/log/ceph/cluster2-client.mirror.54679.log: 89.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54679.log.gz 2023-12-12T02:25:26.275 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51742.log.gz 2023-12-12T02:25:26.295 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50453.log.gz 2023-12-12T02:25:26.327 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49711.log.gz 2023-12-12T02:25:26.360 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50710.log.gz 2023-12-12T02:25:26.371 INFO:teuthology.orchestra.run.smithi140.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53195.log.gz 2023-12-12T02:25:29.921 INFO:teuthology.orchestra.run.smithi055.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2023-12-12T02:25:30.208 INFO:teuthology.orchestra.run.smithi055.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2023-12-12T02:25:30.931 INFO:teuthology.orchestra.run.smithi055.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2023-12-12T02:25:30.932 INFO:teuthology.orchestra.run.smithi055.stderr: 2023-12-12T02:25:30.932 INFO:teuthology.orchestra.run.smithi055.stderr:real 0m8.578s 2023-12-12T02:25:30.932 INFO:teuthology.orchestra.run.smithi055.stderr:user 0m24.885s 2023-12-12T02:25:30.932 INFO:teuthology.orchestra.run.smithi055.stderr:sys 0m0.853s 2023-12-12T02:26:08.390 INFO:teuthology.orchestra.run.smithi140.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2023-12-12T02:26:10.591 INFO:teuthology.orchestra.run.smithi140.stderr: 93.7% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2023-12-12T02:26:15.961 INFO:teuthology.orchestra.run.smithi140.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2023-12-12T02:26:15.962 INFO:teuthology.orchestra.run.smithi140.stderr: 2023-12-12T02:26:15.962 INFO:teuthology.orchestra.run.smithi140.stderr:real 0m53.570s 2023-12-12T02:26:15.962 INFO:teuthology.orchestra.run.smithi140.stderr:user 2m39.069s 2023-12-12T02:26:15.962 INFO:teuthology.orchestra.run.smithi140.stderr:sys 0m6.317s 2023-12-12T02:26:15.963 INFO:tasks.ceph:Archiving logs... 2023-12-12T02:26:15.963 DEBUG:teuthology.misc:Transferring archived files from smithi055:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi055/log 2023-12-12T02:26:15.965 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-12T02:26:18.911 DEBUG:teuthology.misc:Transferring archived files from smithi140:/var/log/ceph to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi140/log 2023-12-12T02:26:18.912 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-12T02:26:39.560 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-12T02:26:39.581 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph.py", line 1928, in task ctx.managers[config['cluster']].wait_for_clean() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2759, in wait_for_clean while not self.is_clean(): File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2695, in is_clean pgs = self.get_pg_stats() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 2356, in get_pg_stats out = self.raw_cluster_cmd('pg', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1611, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_e7f1b79d320b95939278401b2a5bd1b491f81fe3/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_5f3731ee8689ef3b0ef7d571e4cc2cb968c78d73/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-12T02:26:39.582 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-12T02:26:39.582 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-12T02:26:39.585 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-12T02:26:39.653 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-12T02:26:39.700 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-12T02:26:39.700 DEBUG:teuthology.orchestra.run.smithi055:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-12T02:26:39.703 DEBUG:teuthology.orchestra.run.smithi140:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-12T02:26:39.713 INFO:teuthology.orchestra.run.smithi055.stdout: remote refid st t when poll reach delay offset jitter 2023-12-12T02:26:39.713 INFO:teuthology.orchestra.run.smithi055.stdout:============================================================================== 2023-12-12T02:26:39.713 INFO:teuthology.orchestra.run.smithi055.stdout:*hv01.front.sepi 44.4.53.2 2 u 39 128 377 0.091 -2.823 0.439 2023-12-12T02:26:39.714 INFO:teuthology.orchestra.run.smithi055.stdout:+hv02.front.sepi 63.231.80.2 3 u 87 128 377 0.110 -1.733 0.682 2023-12-12T02:26:39.714 INFO:teuthology.orchestra.run.smithi055.stdout:+hv03.front.sepi 74.6.168.72 3 u 81 128 377 0.112 -2.576 0.577 2023-12-12T02:26:39.714 INFO:teuthology.orchestra.run.smithi055.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-12T02:26:39.717 INFO:teuthology.orchestra.run.smithi140.stdout: remote refid st t when poll reach delay offset jitter 2023-12-12T02:26:39.718 INFO:teuthology.orchestra.run.smithi140.stdout:============================================================================== 2023-12-12T02:26:39.718 INFO:teuthology.orchestra.run.smithi140.stdout:*hv01.front.sepi 44.4.53.2 2 u 29 128 377 0.109 -5.812 1.910 2023-12-12T02:26:39.718 INFO:teuthology.orchestra.run.smithi140.stdout:+hv02.front.sepi 63.231.80.2 3 u 101 128 377 0.094 -3.605 1.628 2023-12-12T02:26:39.718 INFO:teuthology.orchestra.run.smithi140.stdout:+hv03.front.sepi 74.6.168.72 3 u 75 128 377 0.110 -4.457 1.487 2023-12-12T02:26:39.718 INFO:teuthology.orchestra.run.smithi140.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2023-12-12T02:26:39.719 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-12T02:26:39.731 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-12T02:26:39.732 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-12T02:26:39.744 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-12T02:26:39.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-12T02:26:39.769 INFO:teuthology.task.internal:Duration was 3264.380536 seconds 2023-12-12T02:26:39.769 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-12T02:26:39.782 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-12T02:26:39.782 DEBUG:teuthology.orchestra.run.smithi055:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T02:26:39.785 DEBUG:teuthology.orchestra.run.smithi140:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-12T02:26:39.822 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-12T02:26:39.822 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi055.front.sepia.ceph.com 2023-12-12T02:26:39.823 DEBUG:teuthology.orchestra.run.smithi055:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T02:26:39.876 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi140.front.sepia.ceph.com 2023-12-12T02:26:39.876 DEBUG:teuthology.orchestra.run.smithi140:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-12T02:26:39.886 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-12T02:26:39.886 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T02:26:39.921 DEBUG:teuthology.orchestra.run.smithi140:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-12T02:26:39.952 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-12T02:26:39.952 DEBUG:teuthology.orchestra.run.smithi055:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T02:26:39.984 DEBUG:teuthology.orchestra.run.smithi140:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-12T02:26:40.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-12T02:26:40.066 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-12T02:26:40.066 DEBUG:teuthology.orchestra.run.smithi055:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T02:26:40.080 DEBUG:teuthology.orchestra.run.smithi140:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-12T02:26:40.090 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-12T02:26:40.103 DEBUG:teuthology.orchestra.run.smithi055:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T02:26:40.124 DEBUG:teuthology.orchestra.run.smithi140:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-12T02:26:40.134 INFO:teuthology.orchestra.run.smithi055.stdout:kernel.core_pattern = core 2023-12-12T02:26:40.144 INFO:teuthology.orchestra.run.smithi140.stdout:kernel.core_pattern = core 2023-12-12T02:26:40.159 DEBUG:teuthology.orchestra.run.smithi055:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T02:26:40.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T02:26:40.207 DEBUG:teuthology.orchestra.run.smithi140:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-12T02:26:40.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-12T02:26:40.212 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-12T02:26:40.226 INFO:teuthology.task.internal:Transferring archived files... 2023-12-12T02:26:40.226 DEBUG:teuthology.misc:Transferring archived files from smithi055:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi055 2023-12-12T02:26:40.227 DEBUG:teuthology.orchestra.run.smithi055:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T02:26:40.269 DEBUG:teuthology.misc:Transferring archived files from smithi140:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100/remote/smithi140 2023-12-12T02:26:40.270 DEBUG:teuthology.orchestra.run.smithi140:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-12T02:26:40.288 INFO:teuthology.task.internal:Removing archive directory... 2023-12-12T02:26:40.288 DEBUG:teuthology.orchestra.run.smithi055:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T02:26:40.308 DEBUG:teuthology.orchestra.run.smithi140:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-12T02:26:40.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-12T02:26:40.346 INFO:teuthology.task.internal:Not uploading archives. 2023-12-12T02:26:40.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-12T02:26:40.441 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-12T02:26:40.441 DEBUG:teuthology.orchestra.run.smithi055:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T02:26:40.444 DEBUG:teuthology.orchestra.run.smithi140:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-12T02:26:40.447 INFO:teuthology.orchestra.run.smithi055.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 02:26 /home/ubuntu/cephtest 2023-12-12T02:26:40.450 INFO:teuthology.orchestra.run.smithi140.stdout: 655472 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 12 02:26 /home/ubuntu/cephtest 2023-12-12T02:26:40.452 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-12T02:26:40.466 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-12T02:26:40.480 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi140 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json' 2023-12-12T02:26:40.481 INFO:teuthology.nuke:Checking targets against current locks 2023-12-12T02:26:40.508 DEBUG:teuthology.nuke:shortname: smithi055 2023-12-12T02:26:40.508 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T02:26:40.533 DEBUG:teuthology.nuke:shortname: smithi140 2023-12-12T02:26:40.533 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-12T02:26:40.536 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi055.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 01:18:22.406121', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIhDAJKfrLVA+K4Ad6CRbRTxf9G11RQdKVLaVWOow2atgabyCvvM6RPcIaCoqkzXKn0+HB/ochBaexdCmpXV9EQ='} 2023-12-12T02:26:40.557 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi140.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/dis-2023-12-12_01:13:29-rbd-wip-dis-testing-1-distro-default-smithi/7488100', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-12 01:18:22.404126', 'locked_by': 'scheduled_dis@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPRz2LD1q4ZzlkncmnwebNaW2K130qCITj6zPV4RfGgzvT1s4SG2i00T3UnQlVQKmsySsbKYfD5Kvf2GQXBRRzs='} 2023-12-12T02:26:40.587 INFO:teuthology.orchestra.console.smithi055:Power off 2023-12-12T02:26:40.587 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T02:26:40.615 DEBUG:teuthology.orchestra.console.smithi055:power off output: Chassis Power Control: Down/Off 2023-12-12T02:26:40.616 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:40.633 INFO:teuthology.orchestra.console.smithi140:Power off 2023-12-12T02:26:40.633 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-12T02:26:40.642 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-12T02:26:40.656 DEBUG:teuthology.orchestra.console.smithi140:power off output: Chassis Power Control: Down/Off 2023-12-12T02:26:40.656 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:40.678 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-12T02:26:44.644 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:44.678 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:44.754 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-12T02:26:44.788 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-12T02:26:48.755 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:48.789 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:48.866 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is on 2023-12-12T02:26:48.899 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is on 2023-12-12T02:26:52.867 DEBUG:teuthology.orchestra.console.smithi055:pexpect command: ipmitool -H smithi055.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:52.900 DEBUG:teuthology.orchestra.console.smithi140:pexpect command: ipmitool -H smithi140.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-12T02:26:52.977 DEBUG:teuthology.orchestra.console.smithi055:check power output: Chassis Power is off 2023-12-12T02:26:53.010 DEBUG:teuthology.orchestra.console.smithi140:check power output: Chassis Power is off 2023-12-12T02:26:53.078 INFO:teuthology.orchestra.console.smithi055:Power off completed 2023-12-12T02:26:53.111 INFO:teuthology.orchestra.console.smithi140:Power off completed 2023-12-12T02:26:53.218 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-minimum} duration: 3264.3805363178253 failure_reason: 'Command failed on smithi140 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json''' flavor: default owner: scheduled_dis@teuthology status: fail success: false 2023-12-12T02:26:53.219 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-12T02:26:53.291 INFO:teuthology.run:FAIL